⌘KCtrl+K
Follow the steps below to download and run the NVIDIA NIM inference microservice for this model on your infrastructure of choice.
Install the NVIDIA GPU Operator
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
&& helm repo update
helm install nim-operator nvidia/k8s-nim-operator --create-namespace -n nim-operator
kubectl create ns nim-service
kubectl create secret -n nim-service docker-registry ngc-secret \
--docker-server=nvcr.io \
--docker-username='$oauthtoken' \
--docker-password=<PASTE_API_KEY_HERE>
kubectl create secret -n nim-service generic ngc-api-secret \
--from-literal=NGC_API_KEY=<PASTE_API_KEY_HERE>
Ensure that a default StorageClass exists in the cluster. If none is present, create an appropriate StorageClass before proceeding.
NOTE:
model-size based on the model and GPU type as described here.nvidia.com/gpu: 1 based on the model and number of GPU requirementsapiVersion: apps.nvidia.com/v1alpha1
kind: NIMService
metadata:
name: nv-embedqa-e5-v5
namespace: nim-service
spec:
image:
repository: nvcr.io/nim/nvidia/nv-embedqa-e5-v5
tag: latest
pullPolicy: IfNotPresent
pullSecrets:
- ngc-secret
authSecret: ngc-api-secret
storage:
pvc:
create: true
size: "model-size"
volumeAccessMode: "ReadWriteOnce"
replicas: 1
resources:
limits:
nvidia.com/gpu: 1
expose:
service:
type: ClusterIP
port: 8000
kubectl run --rm -it -n default curl --image=curlimages/curl:latest -- ash
curl -X "POST" \
'http://nv-embedqa-e5-v5.nim-service:8000/v1/embeddings' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"input": ["Hello world"],
"model": "nvidia/nv-embedqa-e5-v5",
"input_type": "query"
}'
For more details on getting started with this NIM, visit the NVIDIA NIM Operator Docs.