TRT LLM for Inference

Step 1
Configure Docker permissions

To easily manage containers without sudo, you must be in the docker group. If you choose to skip this step, you will need to run Docker commands with sudo.

Open a new terminal and test Docker access. In the terminal, run:

Bash

docker ps

If you see a permission denied error (something like permission denied while trying to connect to the Docker daemon socket), add your user to the docker group so that you don't need to run the command with sudo .

Bash

sudo usermod -aG docker $USER
newgrp docker

Step 2
Verify environment prerequisites

Confirm your Spark device has the required GPU access and network connectivity for downloading models and containers.

Bash

# Check GPU visibility and driver
nvidia-smi

# Verify Docker GPU support
docker run --rm --gpus all nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 nvidia-smi

Step 3
Set environment variables

Bash

# Set `HF_TOKEN` for model access.
export HF_TOKEN=<your-huggingface-token>

export DOCKER_IMAGE="nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13"

Step 4
Validate TensorRT-LLM installation

After confirming GPU access, verify that TensorRT-LLM can be imported inside the container.

Bash

docker run --rm -it --gpus all \
  $DOCKER_IMAGE \
  python -c "import tensorrt_llm; print(f'TensorRT-LLM version: {tensorrt_llm.__version__}')"

Expected output:

Bash

[TensorRT-LLM] TensorRT-LLM version: 1.3.0rc13
TensorRT-LLM version: 1.3.0rc13

Step 5
Create cache directory

Set up local caching to avoid re-downloading models on subsequent runs.

Bash

# Create Hugging Face cache directory
mkdir -p $HOME/.cache/huggingface/

Step 6
Validate setup with quickstart_advanced

This quickstart validates your TensorRT-LLM setup end-to-end by testing model loading, inference engine initialization, and GPU execution with real text generation. It's the fastest way to confirm everything works before starting the inference API server.

LLM quickstart example

Llama 3.1 8B Instruct

Bash

export MODEL_HANDLE="nvidia/Llama-3.1-8B-Instruct-FP4"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
    hf download $MODEL_HANDLE && \
    python examples/llm-api/quickstart_advanced.py \
      --model_dir $MODEL_HANDLE \
      --prompt "Paris is great because" \
      --max_tokens 64
    '

GPT-OSS 20B

Bash

export MODEL_HANDLE="openai/gpt-oss-20b"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
    export TIKTOKEN_ENCODINGS_BASE="/tmp/harmony-reqs" && \
    mkdir -p $TIKTOKEN_ENCODINGS_BASE && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken && \
    hf download $MODEL_HANDLE && \
    python examples/llm-api/quickstart_advanced.py \
      --model_dir $MODEL_HANDLE \
      --prompt "Paris is great because" \
      --max_tokens 64
    '

GPT-OSS 120B

Bash

export MODEL_HANDLE="openai/gpt-oss-120b"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
    export TIKTOKEN_ENCODINGS_BASE="/tmp/harmony-reqs" && \
    mkdir -p $TIKTOKEN_ENCODINGS_BASE && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken && \
    hf download $MODEL_HANDLE && \
    python examples/llm-api/quickstart_advanced.py \
      --model_dir $MODEL_HANDLE \
      --prompt "Paris is great because" \
      --max_tokens 64
    '

Step 7
Validate setup with quickstart_multimodal

VLM quickstart example

This demonstrates vision-language model capabilities by running inference with image understanding. The example uses multimodal inputs to validate both text and vision processing pipelines.

Phi-4-multimodal-instruct

This model requires LoRA (Low-Rank Adaptation) configuration as it uses parameter-efficient fine-tuning. The --load_lora flag enables loading the LoRA weights that adapt the base model for multimodal instruction following.

Bash

export MODEL_HANDLE="nvidia/Phi-4-multimodal-instruct-FP4"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
  python3 examples/llm-api/quickstart_multimodal.py \
    --model_type phi4mm \
    --model_dir $MODEL_HANDLE \
    --modality image \
    --media "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png" \
    --prompt "What is happening in this image?" \
    --load_lora \
    --auto_model_name Phi4MMForCausalLM
  '

NOTE

If you hit a host OOM during downloads or first run, free the OS page cache on the host (outside the container) and retry:

Bash

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

Step 8
Serve LLM with OpenAI-compatible API

Serve with OpenAI-compatible API via trtllm-serve:

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

This example writes nano_v3.yaml for KV cache, MoE, and CUDA graph settings, then starts trtllm-serve on port 8000 with Nemotron Omni reasoning parsers.

Bash

export MODEL_HANDLE="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"

docker run --name trtllm_llm_server --rm -it --gpus all --ipc host --network host \
  -e HF_TOKEN=$HF_TOKEN \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  $DOCKER_IMAGE \
  bash -c '
    hf download $MODEL_HANDLE && \
    cat > nano_v3.yaml <<EOF
kv_cache_config:
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.80
  mamba_ssm_cache_dtype: float32
moe_config:
  backend: CUTLASS
cuda_graph_config:
  enable_padding: true
  max_batch_size: 1
max_batch_size: 1
EOF
    PYTORCH_ALLOC_CONF=expandable_segments:True \
    trtllm-serve serve "$MODEL_HANDLE" \
      --host 0.0.0.0 \
      --port 8355 \
      --trust_remote_code \
      --reasoning_parser nano-v3 \
      --tool_parser qwen3_coder \
      --extra_llm_api_options nano_v3.yaml
  '

Llama 3.1 8B Instruct

Bash

export MODEL_HANDLE="nvidia/Llama-3.1-8B-Instruct-FP4"

docker run --name trtllm_llm_server --rm -it --gpus all --ipc host --network host \
  -e HF_TOKEN=$HF_TOKEN \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  $DOCKER_IMAGE \
  bash -c '
    hf download $MODEL_HANDLE && \
    cat > /tmp/extra-llm-api-config.yml <<EOF
print_iter_log: false
kv_cache_config:
  dtype: "auto"
  free_gpu_memory_fraction: 0.9
cuda_graph_config:
  enable_padding: true
disable_overlap_scheduler: true
EOF
    trtllm-serve "$MODEL_HANDLE" \
      --max_batch_size 64 \
      --trust_remote_code \
      --port 8355 \
      --extra_llm_api_options /tmp/extra-llm-api-config.yml
  '

GPT-OSS 20B

Bash

export MODEL_HANDLE="openai/gpt-oss-20b"

docker run --name trtllm_llm_server --rm -it --gpus all --ipc host --network host \
  -e HF_TOKEN=$HF_TOKEN \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  $DOCKER_IMAGE \
  bash -c '
    export TIKTOKEN_ENCODINGS_BASE="/tmp/harmony-reqs" && \
    mkdir -p $TIKTOKEN_ENCODINGS_BASE && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken && \
    hf download $MODEL_HANDLE && \
    cat > /tmp/extra-llm-api-config.yml <<EOF
print_iter_log: false
kv_cache_config:
  dtype: "auto"
  free_gpu_memory_fraction: 0.9
cuda_graph_config:
  enable_padding: true
disable_overlap_scheduler: true
EOF
    trtllm-serve "$MODEL_HANDLE" \
      --max_batch_size 64 \
      --trust_remote_code \
      --port 8355 \
      --extra_llm_api_options /tmp/extra-llm-api-config.yml
  '

Minimal OpenAI-style chat request. Run this from a separate terminal.

Bash

curl -s http://localhost:8355/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"$MODEL_HANDLE"'",
    "messages": [{"role": "user", "content": "Paris is great because"}],
    "max_tokens": 64
  }'

Step 9
Cleanup and rollback

Remove downloaded models and containers to free up space when testing is complete.

WARNING

This will delete all cached models and may require re-downloading for future runs.

Bash

# Remove Hugging Face cache
sudo chown -R "$USER:$USER" "$HOME/.cache/huggingface"
rm -rf $HOME/.cache/huggingface/

# Clean up Docker images
docker image prune -f
docker rmi $DOCKER_IMAGE

Step 1
Configure Docker permissions

To easily manage containers without sudo, you must be in the docker group. If you choose to skip this step, you will need to run Docker commands with sudo.

Open a new terminal and test Docker access. In the terminal, run:

Bash

docker ps

Bash

sudo usermod -aG docker $USER
newgrp docker

Step 2
Verify environment prerequisites

Confirm your Spark device has the required GPU access and network connectivity for downloading models and containers.

Bash

# Check GPU visibility and driver
nvidia-smi

# Verify Docker GPU support
docker run --rm --gpus all nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 nvidia-smi

Step 3
Set environment variables

Bash

# Set `HF_TOKEN` for model access.
export HF_TOKEN=<your-huggingface-token>

export DOCKER_IMAGE="nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13"

Step 4
Validate TensorRT-LLM installation

After confirming GPU access, verify that TensorRT-LLM can be imported inside the container.

Bash

docker run --rm -it --gpus all \
  $DOCKER_IMAGE \
  python -c "import tensorrt_llm; print(f'TensorRT-LLM version: {tensorrt_llm.__version__}')"

Expected output:

Bash

[TensorRT-LLM] TensorRT-LLM version: 1.3.0rc13
TensorRT-LLM version: 1.3.0rc13

Step 5
Create cache directory

Set up local caching to avoid re-downloading models on subsequent runs.

Bash

# Create Hugging Face cache directory
mkdir -p $HOME/.cache/huggingface/

Step 6
Validate setup with quickstart_advanced

LLM quickstart example

Llama 3.1 8B Instruct

Bash

export MODEL_HANDLE="nvidia/Llama-3.1-8B-Instruct-FP4"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
    hf download $MODEL_HANDLE && \
    python examples/llm-api/quickstart_advanced.py \
      --model_dir $MODEL_HANDLE \
      --prompt "Paris is great because" \
      --max_tokens 64
    '

GPT-OSS 20B

Bash

export MODEL_HANDLE="openai/gpt-oss-20b"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
    export TIKTOKEN_ENCODINGS_BASE="/tmp/harmony-reqs" && \
    mkdir -p $TIKTOKEN_ENCODINGS_BASE && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken && \
    hf download $MODEL_HANDLE && \
    python examples/llm-api/quickstart_advanced.py \
      --model_dir $MODEL_HANDLE \
      --prompt "Paris is great because" \
      --max_tokens 64
    '

GPT-OSS 120B

Bash

export MODEL_HANDLE="openai/gpt-oss-120b"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
    export TIKTOKEN_ENCODINGS_BASE="/tmp/harmony-reqs" && \
    mkdir -p $TIKTOKEN_ENCODINGS_BASE && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken && \
    hf download $MODEL_HANDLE && \
    python examples/llm-api/quickstart_advanced.py \
      --model_dir $MODEL_HANDLE \
      --prompt "Paris is great because" \
      --max_tokens 64
    '

Step 7
Validate setup with quickstart_multimodal

VLM quickstart example

This demonstrates vision-language model capabilities by running inference with image understanding. The example uses multimodal inputs to validate both text and vision processing pipelines.

Phi-4-multimodal-instruct

Bash

export MODEL_HANDLE="nvidia/Phi-4-multimodal-instruct-FP4"

docker run \
  -e MODEL_HANDLE=$MODEL_HANDLE \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  $DOCKER_IMAGE \
  bash -c '
  python3 examples/llm-api/quickstart_multimodal.py \
    --model_type phi4mm \
    --model_dir $MODEL_HANDLE \
    --modality image \
    --media "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png" \
    --prompt "What is happening in this image?" \
    --load_lora \
    --auto_model_name Phi4MMForCausalLM
  '

NOTE

If you hit a host OOM during downloads or first run, free the OS page cache on the host (outside the container) and retry:

Bash

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

Step 8
Serve LLM with OpenAI-compatible API

Serve with OpenAI-compatible API via trtllm-serve:

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

This example writes nano_v3.yaml for KV cache, MoE, and CUDA graph settings, then starts trtllm-serve on port 8000 with Nemotron Omni reasoning parsers.

Bash

export MODEL_HANDLE="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"

docker run --name trtllm_llm_server --rm -it --gpus all --ipc host --network host \
  -e HF_TOKEN=$HF_TOKEN \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  $DOCKER_IMAGE \
  bash -c '
    hf download $MODEL_HANDLE && \
    cat > nano_v3.yaml <<EOF
kv_cache_config:
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.80
  mamba_ssm_cache_dtype: float32
moe_config:
  backend: CUTLASS
cuda_graph_config:
  enable_padding: true
  max_batch_size: 1
max_batch_size: 1
EOF
    PYTORCH_ALLOC_CONF=expandable_segments:True \
    trtllm-serve serve "$MODEL_HANDLE" \
      --host 0.0.0.0 \
      --port 8355 \
      --trust_remote_code \
      --reasoning_parser nano-v3 \
      --tool_parser qwen3_coder \
      --extra_llm_api_options nano_v3.yaml
  '

Llama 3.1 8B Instruct

Bash

export MODEL_HANDLE="nvidia/Llama-3.1-8B-Instruct-FP4"

docker run --name trtllm_llm_server --rm -it --gpus all --ipc host --network host \
  -e HF_TOKEN=$HF_TOKEN \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  $DOCKER_IMAGE \
  bash -c '
    hf download $MODEL_HANDLE && \
    cat > /tmp/extra-llm-api-config.yml <<EOF
print_iter_log: false
kv_cache_config:
  dtype: "auto"
  free_gpu_memory_fraction: 0.9
cuda_graph_config:
  enable_padding: true
disable_overlap_scheduler: true
EOF
    trtllm-serve "$MODEL_HANDLE" \
      --max_batch_size 64 \
      --trust_remote_code \
      --port 8355 \
      --extra_llm_api_options /tmp/extra-llm-api-config.yml
  '

GPT-OSS 20B

Bash

export MODEL_HANDLE="openai/gpt-oss-20b"

docker run --name trtllm_llm_server --rm -it --gpus all --ipc host --network host \
  -e HF_TOKEN=$HF_TOKEN \
  -e MODEL_HANDLE="$MODEL_HANDLE" \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  $DOCKER_IMAGE \
  bash -c '
    export TIKTOKEN_ENCODINGS_BASE="/tmp/harmony-reqs" && \
    mkdir -p $TIKTOKEN_ENCODINGS_BASE && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken && \
    wget -P $TIKTOKEN_ENCODINGS_BASE https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken && \
    hf download $MODEL_HANDLE && \
    cat > /tmp/extra-llm-api-config.yml <<EOF
print_iter_log: false
kv_cache_config:
  dtype: "auto"
  free_gpu_memory_fraction: 0.9
cuda_graph_config:
  enable_padding: true
disable_overlap_scheduler: true
EOF
    trtllm-serve "$MODEL_HANDLE" \
      --max_batch_size 64 \
      --trust_remote_code \
      --port 8355 \
      --extra_llm_api_options /tmp/extra-llm-api-config.yml
  '

Minimal OpenAI-style chat request. Run this from a separate terminal.

Bash

curl -s http://localhost:8355/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"$MODEL_HANDLE"'",
    "messages": [{"role": "user", "content": "Paris is great because"}],
    "max_tokens": 64
  }'

Step 9
Cleanup and rollback

Remove downloaded models and containers to free up space when testing is complete.

WARNING

This will delete all cached models and may require re-downloading for future runs.

Bash

# Remove Hugging Face cache
sudo chown -R "$USER:$USER" "$HOME/.cache/huggingface"
rm -rf $HOME/.cache/huggingface/

# Clean up Docker images
docker image prune -f
docker rmi $DOCKER_IMAGE

TRT LLM for Inference

Step 1
Configure Docker permissions

Step 2
Verify environment prerequisites

Step 3
Set environment variables

Step 4
Validate TensorRT-LLM installation

Step 5
Create cache directory

Step 6
Validate setup with quickstart_advanced

Llama 3.1 8B Instruct

GPT-OSS 20B

GPT-OSS 120B

Step 7
Validate setup with quickstart_multimodal

Phi-4-multimodal-instruct

Step 8
Serve LLM with OpenAI-compatible API

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

Llama 3.1 8B Instruct

GPT-OSS 20B

Step 9
Cleanup and rollback

Resources

TRT LLM for Inference

Step 1
Configure Docker permissions

Step 2
Verify environment prerequisites

Step 3
Set environment variables

Step 4
Validate TensorRT-LLM installation

Step 5
Create cache directory

Step 6
Validate setup with quickstart_advanced

Llama 3.1 8B Instruct

GPT-OSS 20B

GPT-OSS 120B

Step 7
Validate setup with quickstart_multimodal

Phi-4-multimodal-instruct

Step 8
Serve LLM with OpenAI-compatible API

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

Llama 3.1 8B Instruct

GPT-OSS 20B

Step 9
Cleanup and rollback

Resources

TRT LLM for Inference

Step 1Configure Docker permissions

Step 2Verify environment prerequisites

Step 3Set environment variables

Step 4Validate TensorRT-LLM installation

Step 5Create cache directory

Step 6Validate setup with quickstart_advanced

Llama 3.1 8B Instruct

GPT-OSS 20B

GPT-OSS 120B

Step 7Validate setup with quickstart_multimodal

Phi-4-multimodal-instruct

Step 8Serve LLM with OpenAI-compatible API

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

Llama 3.1 8B Instruct

GPT-OSS 20B

Step 9Cleanup and rollback

Resources

TRT LLM for Inference

Step 1Configure Docker permissions

Step 2Verify environment prerequisites

Step 3Set environment variables

Step 4Validate TensorRT-LLM installation

Step 5Create cache directory

Step 6Validate setup with quickstart_advanced

Llama 3.1 8B Instruct

GPT-OSS 20B

GPT-OSS 120B

Step 7Validate setup with quickstart_multimodal

Phi-4-multimodal-instruct

Step 8Serve LLM with OpenAI-compatible API

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

Llama 3.1 8B Instruct

GPT-OSS 20B

Step 9Cleanup and rollback

Resources

Step 1
Configure Docker permissions

Step 2
Verify environment prerequisites

Step 3
Set environment variables

Step 4
Validate TensorRT-LLM installation

Step 5
Create cache directory

Step 6
Validate setup with quickstart_advanced

Step 7
Validate setup with quickstart_multimodal

Step 8
Serve LLM with OpenAI-compatible API

Step 9
Cleanup and rollback

Step 1
Configure Docker permissions

Step 2
Verify environment prerequisites

Step 3
Set environment variables

Step 4
Validate TensorRT-LLM installation

Step 5
Create cache directory

Step 6
Validate setup with quickstart_advanced

Step 7
Validate setup with quickstart_multimodal

Step 8
Serve LLM with OpenAI-compatible API

Step 9
Cleanup and rollback