Blueprint Overview

Use Case Description

Insightful, accurate, and interactive video analytics AI agents enable a range of industries to make better decisions faster. These AI agents are given tasks through natural language and can perform complex operations like video summarization and visual question-answering, unlocking entirely new application possibilities. The NVIDIA AI Blueprint makes it easy to get started building and customizing video analytics AI agents for video search and summarization — all powered by generative AI, vision language models (VLMs) like Cosmos Nemotron VLMs, large language models (LLMs) like Llama Nemotron LLMs, NVIDIA NeMo Retriever, and NVIDIA NIM.

Architecture Diagram

Included NIM

The following NIM microservices are used in this blueprint:

cosmos-nemotron-34b
meta / llama-3.1-70b-instruct
llama-3_2-nv-embedqa-1b-v2
llama-3_2-nv-rerankqa-1b-v2

Minimum System requirements

Core engine

The core video search and summarization blueprint pipeline supports the following hardware:

• A6000
• L40/L40s
• A100
• H100

Hosted NIMs

• Llama-3.1-70b-instruct requires the following minimum GPU configuration based on this support matrix.
• Cosmos Nemotron 34b VLM requires 1xA100 as a minimum GPU.
• NeMo Retriever Embedding NIM requires 1xL40S as a minimum GPU.
• NeMo Reranking Embedding NIM requires 1xL40S as a minimum GPU.

What’s included in the Blueprint

NVIDIA AI Blueprints are customizable agentic workflow examples that equip enterprise developers with NIM microservices, reference code, documentation, and a Helm chart for deployment. This blueprint offers users a reference architecture for deploying a visual agent adept at understanding long-form videos. The agent integrates a scalable video ingestion pipeline, a context manager, a vision-language model (VLM), a large language model (LLM), and a Context-Aware Retrieval-Augmented Generation (CA-RAG) module. CA-RAG module leverages dense captions stored in vector and graph databases as its primary sources for video understanding.

Key Agent Features

• Summarization
• Interactive Question and Answering (Q&A)
• Alerts (Not in Preview Experience)

Core Technology

• Data processing pipeline: The process involves decoding video segments (chunks) generated by the stream handler, selecting frames, and using a vision-language model (VLM) along with a caption prompt to generate detailed captions for each chunk. These dense captions are then indexed into vector and graph databases for use in the Context-Aware Retrieval-Augmented Generation workflow.
• Context Manager: Efficiently incorporates tools—a vision-language model (VLM) and a large language model (LLM), using them as required—and key functions including a summary generator, an answer generator, and an alert handler. The tools and functions are used in summary generation, handling Q&A, and managing alerts. In addition, context manager effectively maintains its working context by making efficient use of both short-term memory, such as chat history, and long-term memory resources like vector and graph databases, as needed.
• CA-RAG module: The Context-Aware Retrieval-Augmented Generation (CA-RAG) module leverages both Vector RAG and Graph-RAG as the primary sources for video understanding. During the Q&A workflow, the CA-RAG module extracts relevant context from the vector database and graph database to enhance temporal reasoning, anomaly detection, multi-hop reasoning, and scalability, thereby offering deeper contextual understanding and efficient management of extensive video data.
• NeMo Guardrails: Filters out invalid user prompts. It makes use of the REST API of an LLM NIM microservice.

Example Walkthrough

The user selects an example video and prompt to guide the agent in generating a detailed summary. The agent splits the input video into smaller segments that are processed by a VLM (The preview uses OpenAI's GPT4o). These segments are processed in parallel by the VLM pipeline to produce detailed captions describing the events of each chunk in a scalable and efficient manner. The agent recursively summarizes the dense captions using an LLM, generating a final summary for the entire video once all chunk captions are processed.

Additionally, these captions are stored in vector and graph databases to power the Q&A feature of this blueprint, allowing the user to ask any open-ended questions about the video.

License

Use of the models in this blueprint is governed by the NVIDIA AI Foundation Models Community License.

Terms of Use

GOVERNING TERMS: This preview is governed by the NVIDIA API Trial Terms of Service.

Additional Information:

For the model that includes a Llama3.1 model: Llama 3.1 Community License Agreement, Built with Llama.

For the NVIDIA Retrieval QA Mistral 4B Reranking: Apache license.

For the NVIDIA Retrieval QA E5 Embedding v5: NV-EmbedQA-E5-v5: MIT license; NV-EmbedQA-Mistral7B-v2: Apache 2.0 license, and Snowflake arctic-embed-l: Apache 2.0 license.