Build a Video Search and Summarization (VSS) Agent

Step 1
Verify environment requirements

Check that your system meets the hardware and software prerequisites.

Bash

# Verify driver version
nvidia-smi | grep "Driver Version"
# Expected output: Driver Version: 580.95.05 or higher

# Verify CUDA version
nvcc --version
# Expected output: release 13.0

# Verify Docker is running
docker --version && docker compose version

Step 2
Configure Docker

To easily manage containers without sudo, you must be in the docker group. If you choose to skip this step, you will need to run Docker commands with sudo. Open a new terminal and test Docker access. In the terminal, run:

Bash

docker ps

If you see a permission denied error (something like permission denied while trying to connect to the Docker daemon socket), add your user to the docker group so that you don't need to run the command with sudo .

Bash

sudo usermod -aG docker $USER
newgrp docker

Additionally, configure Docker so that it can use the NVIDIA Container Runtime.

Bash

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

#Run a sample workload to verify the setup
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Step 3
Clone the VSS repository

Clone the Video Search and Summarization repository from NVIDIA's public GitHub.

Note Install Git LFS if not already present on the system

Bash

sudo apt-get install -y git-lfs && git lfs install

Bash

# Clone the VSS AI Blueprint repository
git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout tags/v3.2.0
git lfs install
git lfs pull

Step 4
Run the cache cleaner script

Start the system cache cleaner to optimize memory usage during container operations.

Create the cache cleaner script at /usr/local/bin/sys-cache-cleaner.sh mentioned below

Bash

sudo tee /usr/local/bin/sys-cache-cleaner.sh << 'EOF'
#!/bin/bash
# Exit immediately if any command fails
set -e

# Disable hugepages
echo "disable vm/nr_hugepage"
echo 0 | tee /proc/sys/vm/nr_hugepages

# Notify that the cache cleaner is running
echo "Starting cache cleaner - Running"
echo "Press Ctrl + C to stop"
# Repeatedly sync and drop caches every 3 seconds
while true; do
     sync && echo 3 | tee /proc/sys/vm/drop_caches > /dev/null
     sleep 3
done
EOF

sudo chmod +x /usr/local/bin/sys-cache-cleaner.sh

Running in the background

Bash

# In another terminal, start the cache cleaner script.
sudo -b /usr/local/bin/sys-cache-cleaner.sh

NOTE

The above runs the cache cleaner in the current session only; it does not persist across reboots. To have the cache cleaner run across reboots, create a systemd service instead. To stop the background cache cleaner:

Bash

sudo pkill -f sys-cache-cleaner.sh

Step 5
Authenticate with NVIDIA Container Registry

NOTE

If you don’t have an NVIDIA account already, you’ll have to create one and register for the developer program.

Bash

# Log in to NVIDIA Container Registry
docker login nvcr.io
# Username: $oauthtoken
# Password: <PASTE_NGC_API_KEY_HERE>

Step 6
Choose deployment scenario

Choose the deployment options based on your requirements:

Deployment Scenario	VLM (Cosmos-Reason2-8B)	LLM
Standard VSS (Base)	Local	Remote
Standard VSS (Alert Verification)	Local	Remote
Standard VSS deployment (Real-Time Alerts)	Local	Remote

Step 7
Standard VSS

Standard VSS (Hybrid Deployment)

In this hybrid deployment, we would use NIMs from build.nvidia.com. Alternatively, you can configure your own hosted endpoints by following the instructions in the VSS remote LLM deployment guide.

7.1 Get NVIDIA API Key

Log in to https://build.nvidia.com/explore/discover.
Search for Get API Key on the page and click on it.

7.2 Launch Standard VSS deployment

Standard VSS deployment (Base) Standard VSS deployment (Alert Verification) Standard VSS deployment (Real-Time Alerts)

Bash

# Start Standard VSS (Base)
# Set NGC CLI API key and Hugging Face token (required for VA-MCP)
export NGC_CLI_API_KEY='your_ngc_api_key'
export HF_TOKEN='hf_your_token_here'
export LLM_ENDPOINT_URL=https://your-llm-endpoint.com
deploy/docker/scripts/dev-profile.sh up -p base -H DGX-SPARK --use-remote-llm --llm <REMOTE LLM MODEL NAME>

# Start Standard VSS (Alert Verification)
export NGC_CLI_API_KEY='your_ngc_api_key'
export LLM_ENDPOINT_URL=https://your-llm-endpoint.com
deploy/docker/scripts/dev-profile.sh up -p alerts -m verification -H DGX-SPARK --use-remote-llm --llm <REMOTE LLM MODEL NAME>

# Start Standard VSS (Real-Time Alerts)
export NGC_CLI_API_KEY='your_ngc_api_key'
export LLM_ENDPOINT_URL=https://your-llm-endpoint.com
deploy/docker/scripts/dev-profile.sh up -p alerts -m real-time -H DGX-SPARK --use-remote-llm --llm <REMOTE LLM MODEL NAME>

NOTE

This step will take several minutes as containers are pulled and services initialize. The VSS backend requires additional startup time. Set the following environment variables before deployment: • NGC_CLI_API_KEY — (required) NGC API key for pulling images and deployment • LLM_ENDPOINT_URL — (required when using --use-remote-llm) Base URL for the remote LLM • NVIDIA_API_KEY — (optional) For remote LLM/VLM endpoints that require it • OPENAI_API_KEY — (optional) For remote LLM/VLM endpoints that require it • VLM_CUSTOM_WEIGHTS — (optional) Absolute path to a custom weights directory

Pass these additional flags to deploy/docker/scripts/dev-profile.sh for remote LLM mode: • --use-remote-llm — (required) Use a remote LLM, the base URL is read from LLM_ENDPOINT_URL in the environment • --llm — (required) Remote LLM model name (for example: nvidia/nvidia-nemotron-nano-9b-v2). Strongly recommended for alert workflows (verification and real-time): use nvidia/nvidia-nemotron-nano-9b-v2. Omitting --llm may cause the script to use whatever model is returned by the remote endpoint.

Run deploy/docker/scripts/dev-profile.sh --help for a full list of supported arguments.

7.3 Validate Standard VSS deployment

Access the VSS UI to confirm successful deployment. Common VSS Endpoints

Bash

# Test Agent UI accessibility
# If running locally on your Spark device, use localhost:
curl -I http://localhost:7777
# Expected: HTTP 200 response

# If your Spark is running in Remote/Accessory mode, replace 'localhost' with the IP address or hostname of your Spark device.
# To find your Spark's IP address, run the following command on the Spark terminal:
hostname -I
# Or to get the hostname:
hostname
# Then test accessibility (replace <SPARK_IP_OR_HOSTNAME> with the actual value):
curl -I http://<SPARK_IP_OR_HOSTNAME>:7777

Open http://localhost:7777 or http://<SPARK_IP_OR_HOSTNAME>:7777 in your browser to access the Agent interface.

Step 8
Test video processing workflow

Run a basic test to verify the video analysis pipeline is functioning based on your deployment.

For Standard VSS deployment

Follow the steps here to navigate VSS Agent UI.

Access VSS Agent interface at http://localhost:7777
Download the sample data from NGC here and upload videos and test features
Test Standard VSS deployment (Base) here
Test Standard VSS deployment (Alert Verification) here
Test Standard VSS deployment (Real-Time Alerts) here

Step 9
Cleanup and rollback

To completely remove the VSS deployment and free up system resources Follow:

WARNING

This will destroy all processed video data and analysis results.

Bash

# For Standard VSS deployment
deploy/docker/scripts/dev-profile.sh down

Step 10
Next steps

With VSS deployed, you can now:

Standard VSS deployment:

Access full VSS capabilities at port 7777
Test video and Q&A features
Configure knowledge graphs and graph databases
Integrate with existing video processing workflows

Step 1
Verify environment requirements

Check that your system meets the hardware and software prerequisites.

Bash

# Verify driver version
nvidia-smi | grep "Driver Version"
# Expected output: Driver Version: 580.95.05 or higher

# Verify CUDA version
nvcc --version
# Expected output: release 13.0

# Verify Docker is running
docker --version && docker compose version

Step 2
Configure Docker

Bash

docker ps

Bash

sudo usermod -aG docker $USER
newgrp docker

Additionally, configure Docker so that it can use the NVIDIA Container Runtime.

Bash

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

#Run a sample workload to verify the setup
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Step 3
Clone the VSS repository

Clone the Video Search and Summarization repository from NVIDIA's public GitHub.

Note Install Git LFS if not already present on the system

Bash

sudo apt-get install -y git-lfs && git lfs install

Bash

# Clone the VSS AI Blueprint repository
git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout tags/v3.2.0
git lfs install
git lfs pull

Step 4
Run the cache cleaner script

Start the system cache cleaner to optimize memory usage during container operations.

Create the cache cleaner script at /usr/local/bin/sys-cache-cleaner.sh mentioned below

Bash

sudo tee /usr/local/bin/sys-cache-cleaner.sh << 'EOF'
#!/bin/bash
# Exit immediately if any command fails
set -e

# Disable hugepages
echo "disable vm/nr_hugepage"
echo 0 | tee /proc/sys/vm/nr_hugepages

# Notify that the cache cleaner is running
echo "Starting cache cleaner - Running"
echo "Press Ctrl + C to stop"
# Repeatedly sync and drop caches every 3 seconds
while true; do
     sync && echo 3 | tee /proc/sys/vm/drop_caches > /dev/null
     sleep 3
done
EOF

sudo chmod +x /usr/local/bin/sys-cache-cleaner.sh

Running in the background

Bash

# In another terminal, start the cache cleaner script.
sudo -b /usr/local/bin/sys-cache-cleaner.sh

NOTE

Bash

sudo pkill -f sys-cache-cleaner.sh

Step 5
Authenticate with NVIDIA Container Registry

NOTE

If you don’t have an NVIDIA account already, you’ll have to create one and register for the developer program.

Bash

# Log in to NVIDIA Container Registry
docker login nvcr.io
# Username: $oauthtoken
# Password: <PASTE_NGC_API_KEY_HERE>

Step 6
Choose deployment scenario

Choose the deployment options based on your requirements:

Deployment Scenario	VLM (Cosmos-Reason2-8B)	LLM
Standard VSS (Base)	Local	Remote
Standard VSS (Alert Verification)	Local	Remote
Standard VSS deployment (Real-Time Alerts)	Local	Remote

Step 7
Standard VSS

Standard VSS (Hybrid Deployment)

In this hybrid deployment, we would use NIMs from build.nvidia.com. Alternatively, you can configure your own hosted endpoints by following the instructions in the VSS remote LLM deployment guide.

7.1 Get NVIDIA API Key

Log in to https://build.nvidia.com/explore/discover.
Search for Get API Key on the page and click on it.

7.2 Launch Standard VSS deployment

Standard VSS deployment (Base) Standard VSS deployment (Alert Verification) Standard VSS deployment (Real-Time Alerts)

Bash

# Start Standard VSS (Base)
# Set NGC CLI API key and Hugging Face token (required for VA-MCP)
export NGC_CLI_API_KEY='your_ngc_api_key'
export HF_TOKEN='hf_your_token_here'
export LLM_ENDPOINT_URL=https://your-llm-endpoint.com
deploy/docker/scripts/dev-profile.sh up -p base -H DGX-SPARK --use-remote-llm --llm <REMOTE LLM MODEL NAME>

# Start Standard VSS (Alert Verification)
export NGC_CLI_API_KEY='your_ngc_api_key'
export LLM_ENDPOINT_URL=https://your-llm-endpoint.com
deploy/docker/scripts/dev-profile.sh up -p alerts -m verification -H DGX-SPARK --use-remote-llm --llm <REMOTE LLM MODEL NAME>

# Start Standard VSS (Real-Time Alerts)
export NGC_CLI_API_KEY='your_ngc_api_key'
export LLM_ENDPOINT_URL=https://your-llm-endpoint.com
deploy/docker/scripts/dev-profile.sh up -p alerts -m real-time -H DGX-SPARK --use-remote-llm --llm <REMOTE LLM MODEL NAME>

NOTE

Run deploy/docker/scripts/dev-profile.sh --help for a full list of supported arguments.

7.3 Validate Standard VSS deployment

Access the VSS UI to confirm successful deployment. Common VSS Endpoints

Bash

# Test Agent UI accessibility
# If running locally on your Spark device, use localhost:
curl -I http://localhost:7777
# Expected: HTTP 200 response

# If your Spark is running in Remote/Accessory mode, replace 'localhost' with the IP address or hostname of your Spark device.
# To find your Spark's IP address, run the following command on the Spark terminal:
hostname -I
# Or to get the hostname:
hostname
# Then test accessibility (replace <SPARK_IP_OR_HOSTNAME> with the actual value):
curl -I http://<SPARK_IP_OR_HOSTNAME>:7777

Open http://localhost:7777 or http://<SPARK_IP_OR_HOSTNAME>:7777 in your browser to access the Agent interface.

Step 8
Test video processing workflow

Run a basic test to verify the video analysis pipeline is functioning based on your deployment.

For Standard VSS deployment

Follow the steps here to navigate VSS Agent UI.

Access VSS Agent interface at http://localhost:7777
Download the sample data from NGC here and upload videos and test features
Test Standard VSS deployment (Base) here
Test Standard VSS deployment (Alert Verification) here
Test Standard VSS deployment (Real-Time Alerts) here

Step 9
Cleanup and rollback

To completely remove the VSS deployment and free up system resources Follow:

WARNING

This will destroy all processed video data and analysis results.

Bash

# For Standard VSS deployment
deploy/docker/scripts/dev-profile.sh down

Step 10
Next steps

With VSS deployed, you can now:

Standard VSS deployment:

Access full VSS capabilities at port 7777
Test video and Q&A features
Configure knowledge graphs and graph databases
Integrate with existing video processing workflows

Build a Video Search and Summarization (VSS) Agent

Step 1
Verify environment requirements

Step 2
Configure Docker

Step 3
Clone the VSS repository

Step 4
Run the cache cleaner script

Step 5
Authenticate with NVIDIA Container Registry

Step 6
Choose deployment scenario

Step 7
Standard VSS

Step 8
Test video processing workflow

Step 9
Cleanup and rollback

Step 10
Next steps

Resources

Build a Video Search and Summarization (VSS) Agent

Step 1
Verify environment requirements

Step 2
Configure Docker

Step 3
Clone the VSS repository

Step 4
Run the cache cleaner script

Step 5
Authenticate with NVIDIA Container Registry

Step 6
Choose deployment scenario

Step 7
Standard VSS

Step 8
Test video processing workflow

Step 9
Cleanup and rollback

Step 10
Next steps

Resources

Build a Video Search and Summarization (VSS) Agent

Step 1Verify environment requirements

Step 2Configure Docker

Step 3Clone the VSS repository

Step 4Run the cache cleaner script

Step 5Authenticate with NVIDIA Container Registry

Step 6Choose deployment scenario

Step 7Standard VSS

Step 8Test video processing workflow

Step 9Cleanup and rollback

Step 10Next steps

Resources

Build a Video Search and Summarization (VSS) Agent

Step 1Verify environment requirements

Step 2Configure Docker

Step 3Clone the VSS repository

Step 4Run the cache cleaner script

Step 5Authenticate with NVIDIA Container Registry

Step 6Choose deployment scenario

Step 7Standard VSS

Step 8Test video processing workflow

Step 9Cleanup and rollback

Step 10Next steps

Resources

Step 1
Verify environment requirements

Step 2
Configure Docker

Step 3
Clone the VSS repository

Step 4
Run the cache cleaner script

Step 5
Authenticate with NVIDIA Container Registry

Step 6
Choose deployment scenario

Step 7
Standard VSS

Step 8
Test video processing workflow

Step 9
Cleanup and rollback

Step 10
Next steps

Step 1
Verify environment requirements

Step 2
Configure Docker

Step 3
Clone the VSS repository

Step 4
Run the cache cleaner script

Step 5
Authenticate with NVIDIA Container Registry

Step 6
Choose deployment scenario

Step 7
Standard VSS

Step 8
Test video processing workflow

Step 9
Cleanup and rollback

Step 10
Next steps