Nemotron Voice Agent Blueprint by NVIDIA

Voice is the most natural interface for efficient and high-speed communication. This developer example provides a comprehensive, end-to-end voice agent blueprint, built with NVIDIA Nemotron state-of-the-art open models, as NVIDIA NIM for acceleration and scaling. It is designed to guide developers through the creation of a cascaded pipeline, integrating Nemotron automatic speech recognition (ASR), large language model (LLM), and text-to-speech (TTS), while solving for the complexities of streaming, interruptible (barge-in) conversations. By leveraging NVIDIA NIM, this developer example enables developers to accelerate the deployment of high-performance voice AI solutions. Clone it, swap in your own logic, and try this prototype - deployable voice AI in hours.

Architecture Diagram

Key Features

Advanced Interruption Management: Features built-in Voice Activation Detection (VAD) and End of Utterance (EOU) logic to guide the agent on exactly when to start and stop speaking, ensuring a natural conversational flow.
Multilingual Capabilities: Native support for the 9 languages optimized by NVIDIA Magpie TTS Multilingual.
Multimodal Understanding: Reason over speech and vision together — analyze live camera input and uploaded media (images, documents) within a single conversation, powered by Nemotron Omni Model.
Multi-Agent & Tool Calling: Orchestrate cooperating agents that invoke external tools and functions for task-oriented workflows, while decoupling reasoning from response generation for lower perceived latency.
Enterprise Scaling: Supports multiple concurrent instances.
Edge Support: Deploy anywhere, from cloud and workstation to DGX Spark and edge devices like Jetson Thor, using self-contained deployment recipes.

Benchmarks

Model / API	Reasoning Mode	Text Only Standalone LLM (%)	LLM In Voice Agent Pipeline (%)
Llama 3.3 Nemotron Super 49B	Reasoning ON	91.90	81.30
Llama 3.3 Nemotron Super 49B	Reasoning OFF	82.70	60.30
Nemotron 3 Nano 30B	Reasoning ON Reasoning Budget - 500	78.76	75.60
Nemotron 3 Nano 30B	Reasoning OFF	56.50	50.40

Benchmarks on the Big Bench Audio dataset based on internal testing. Evaluation source code provided on GitHub.

Parallel Streams	E2E Latency	ASR Latency	TTS TTFB	LLM TTFT	LLM first-sentence latency
1	0.79	0.04	0.078	0.126	0.138
4	0.76	0.046	0.066	0.061	0.181
8	0.77	0.052	0.066	0.062	0.136
16	0.91	0.057	0.068	0.105	0.208
32	0.8	0.061	0.08	0.073	0.294
64	1	0.067	0.11	0.156	0.386

The benchmark table demonstrates that the NVIDIA Nemotron Voice Agent achieves sub-second End-to-End Latency across up to 64 parallel streams with a setup utilizing 4xH100 GPUs (one for Parakeet CTC 1.1B, one for Magpie TTS, and two for Nemotron-3-Nano LLM model) with speculative speech processing enabled.

This developer example is powered by a suite of NVIDIA-optimized microservices designed for maximum throughput and minimal latency.

Category	Component	Recommended Model
Speech-to-Text or Automatic Speech Recognition	ASR / AST	NVIDIA Nemotron Speech ASR (RNNT or CTC)
Logic & Reasoning	LLM	Nemotron 3 Nano / Nemotron 3 Super / Nemotron 3 Omni
Text-to-Speech	TTS	Magpie TTS Multilingual
Control	Behavioral Logic	VAD, SVAD, EOU

Self-Hosted Configuration

To achieve sub-second response times and high-fidelity audio handling, the following hardware configurations are recommended for local deployment.

Service	Use Case	Recommended GPU
Nemotron Speech ASR/TTS	Audio Transcription & Synthesis	1x L40, A100 (80GB), or H100
Reasoning Model	LLM & Agentic Logic	2x H100 (80GB) or 4x A100 (80GB)
Voice Agent	Entire workflow on local or edge deployments	DGX Spark, Local Workstation( 1x GPU with 72 GB+ VRAM), Jetson Thor

Use Cases

Healthcare: A voice-first agent automates the patient intake process by capturing symptoms and medical history hands-free, allowing clinicians to focus entirely on patient care.
Telco: Real-time virtual assistants handle complex technical troubleshooting and plan upgrades over the phone, reducing wait times while providing instant, multilingual support.
Retail: Interactive voice agents integrated into in-store kiosks or mobile apps allow customers to browse inventory, check prices, and manage their carts using natural conversation.
Financial Services (FSI): Secure voice agents enable customers to perform balance inquiries, confirm recent payments, or initiate fraud disputes through a low-latency voice interface.
Airlines: Travel assistants provide immediate support for rebooking flights, tracking baggage, or selecting seats during high-volume disruption events when traditional call centers are overwhelmed.
Hospitality: Digital concierge agents handle room service orders and local recommendations in multiple languages, providing a premium, high-speed guest experience without human intervention.

Getting Started

The code base serves as a playground to test new models and expand basic ASR/LLM/TTS flows.

Clone the Repo: Access the public reference code on GitHub.

Setup NVIDIA NIM: Deploy your local system/pipeline using NVIDIA NIM microservices.

Pick an example: Deploy with a single Docker Compose command. Available examples:

Example	Description
Generic Assistant	Cascaded ASR, LLM, TTS pipeline (English)
Multilingual Assistant	Multilingual ASR with automatic TTS language switching
Omni Assistant	Omni model as a replacement for ASR + LLM with Magpie TTS
Omni Assistant Subagents	Multi-agent with live camera vision and attachment analysis
Frontend Backend Agent	Frontend conversation agent paired with a backend airline agent for tool calling

The examples empower developers to rapidly build, customize, and deploy enterprise-grade voice agents for customer service, and user interactions in healthcare, telecom, retail, and financial services.

Explore the Ambient Healthcare Agents blueprint to deploy ambient agents that assist with patient intake, symptom triage, and compliance for HIPAA/PCI. This blueprint has tightly integrated, healthcare-tuned models (e.g., clinical LLMs, medical diarization, guardrails for HIPAA alignment, SOAP/ICD form automation). It is designed to be “out-of-the-box” for clinical scenarios.

Additional Voice Agent Examples

Scalable Voice-to-Voice Workflow: Production reference Kubernetes deployment using NVIDIA NIM for optimized inference, featuring custom Prometheus and Grafana observability. GitHub Repository
Integrated ASR+EOU, tracking different speakers across turns, tool calling, evaluation pipeline: GitHub Repository
Daily/Pipecat - This repo is sample code for building voice agents with three NVIDIA open source models: Nemotron Speech ASR, Nemotron 3 Nano LLM, Magpie TTS (Preview) GitHub Repository. Read the blog post for more details. Deployed on DGX Spark.
Vertical examples - five examples from different verticals to converse with a voice agent - https://github.com/fciannella/nemotron-speech-demos/tree/main .
1. Claim Investigator (Tier 1)
2. Healthcare Agent
3. Banking Fee Inquiry Resolver
4. Telco Agent: Billing Specialist
5. Wire Transfer Agent with Twilio integration. (Call transfer) included.
Ambient Healthcare Agents + Guardrails - Github Repository

Ethical considerations

NVIDIA believes trustworthy AI is a shared responsibility. When using this example in accordance with our terms of service, work with your model and compliance teams to ensure the system meets requirements for your industry and use case. Report security or AI concerns here.