Nemotron Voice Agent Blueprint by NVIDIA

Voice is the most natural interface for efficient and high-speed communication. This developer example provides a comprehensive, end-to-end voice agent blueprint, built with NVIDIA Nemotron state-of-the-art open models, as NVIDIA NIM for acceleration and scaling. It is designed to guide developers through the creation of a cascaded pipeline, integrating Nemotron automatic speech recognition (ASR), large language model (LLM), and text-to-speech (TTS), while solving for the complexities of streaming, interruptible (barge-in) conversations. By leveraging NVIDIA NIM, this developer example enables developers to accelerate the deployment of high-performance voice AI solutions. Clone it, swap in your own logic, and try this prototype - deployable voice AI in hours.

Architecture Diagram

Key Features

Advanced Interruption Management: Features built-in Voice Activation Detection (VAD) and End of Utterance (EOU) logic to guide the agent on exactly when to start and stop speaking, ensuring a natural conversational flow.
Multilingual Capabilities: Native support for the 12 languages optimized by NVIDIA Magpie TTS Multilingual.
Multimodal Understanding: Reason over speech and vision together — analyze live camera input and uploaded media (images, documents) within a single conversation, powered by Nemotron Omni Model.
Multi-Agent & Tool Calling: Orchestrate cooperating agents that invoke external tools and functions for task-oriented workflows, while decoupling reasoning from response generation for lower perceived latency.
Enterprise Scaling: Supports high-concurrency and multi-instance deployments using NVIDIA NIMs.
Open Models and Open Inference: Nemotorn family of open models and inference using Nemo-speech.cpp for speech models and vLLM for LLM models.
Edge Support: Deploy anywhere, from cloud and workstation to DGX Spark and edge devices like Jetson Thor, using self-contained deployment recipes.
Agent Skills: Use nemotron-voice-agent-builder skill to quickly scaffold and refine voice agent pipeline with NVIDIA models using Pipecat or LiveKit.

Benchmarks

Parallel Streams	E2E Latency	ASR + Turn Taking Latency	TTS TTFB	LLM TTFT	LLM tokens per second
1	0.93	0.47	0.08	0.23	144
4	0.86	0.49	0.07	0.17	170
8	0.87	0.49	0.08	0.14	161
16	0.86	0.49	0.08	0.13	154
32	0.9	0.48	0.09	0.13	143
64	0.93	0.49	0.1	0.13	141

The benchmark table demonstrates that the NVIDIA Nemotron Voice Agent achieves sub-second End-to-End Latency across up to 64 parallel streams with a setup utilizing 4xB200 GPUs (one for Nemotron Streaming ASR 0.6B, one for Magpie TTS, and two for Nemotron 3.5 Lightning LLM model).

Turn strategy	Pause Handling		Backchannel			Smooth Turn Taking		User Interruption
	Synthetic TOR ↓	Candor TOR ↓	TOR ↓	Freq ↑	JSD ↓	Candor TOR ↑	Latency ↓	TOR ↑	Latency ↓
Pipecat Smart turn strategy with 1 sec timeout (default)	0.277	0.574	0.509	0.038	0.847	0.983	1.044	0.985	1.97
silero vad 500ms	0.58	0.82	0.73	0.1	0.72	0.99	0.62	0.99	0.91

Full-Duplex-Bench 1.0 evaluates how naturally full-duplex voice agents manage real-time conversations. The table compares the pipecat smart turn strategy 1 sec timeout with a 500 ms Silero VAD baseline across pause handling, backchanneling, smooth turn-taking, and user interruptions. Arrows indicate whether lower or higher values represent better performance.

This developer example is powered by a suite of NVIDIA-optimized microservices designed for maximum throughput and minimal latency.

Category	Component	Recommended Model
Speech-to-Text or Automatic Speech Recognition	ASR / AST	NVIDIA Nemotron Speech ASR (RNNT or CTC)
Logic & Reasoning	LLM	Nemotron 3.5 Lightning / Nemotron 3 Super / Nemotron 3 Omni
Text-to-Speech	TTS	Magpie TTS Multilingual
Control	Behavioral Logic	VAD, SVAD, EOU

Self-Hosted Configuration

To achieve sub-second response times and high-fidelity audio handling, the following hardware configurations are recommended for local deployment.

Service	Use Case	Recommended GPU
Nemotron Speech ASR/TTS	Audio Transcription & Synthesis	1x L40, H100, or B200
Reasoning Model	LLM & Agentic Logic	2x H100 (80GB) or 1x B200
Voice Agent	Entire workflow on local or edge deployments	DGX Spark, Local Workstation(1x GPU with 48 GB+ VRAM), Jetson Thor

Use Cases

Healthcare: A voice-first agent automates the patient intake process by capturing symptoms and medical history hands-free, allowing clinicians to focus entirely on patient care.
Telco: Real-time virtual assistants handle complex technical troubleshooting and plan upgrades over the phone, reducing wait times while providing instant, multilingual support.
Retail: Interactive voice agents integrated into in-store kiosks or mobile apps allow customers to browse inventory, check prices, and manage their carts using natural conversation.
Financial Services (FSI): Secure voice agents enable customers to perform balance inquiries, confirm recent payments, or initiate fraud disputes through a low-latency voice interface.
Airlines: Travel assistants provide immediate support for rebooking flights, tracking baggage, or selecting seats during high-volume disruption events when traditional call centers are overwhelmed.
Hospitality: Digital concierge agents handle room service orders and local recommendations in multiple languages, providing a premium, high-speed guest experience without human intervention.

Getting Started

The code base serves as a playground to test new models and expand basic ASR/LLM/TTS flows.

Clone the Repo: Access the public reference code on GitHub.

Setup NVIDIA NIM: Deploy your local system/pipeline using NVIDIA NIM microservices.

Pick an example: Deploy with a single Docker Compose command. Available examples:

Example	Description
Generic Assistant	Cascaded ASR, LLM, TTS pipeline (English)
Multilingual Assistant	Multilingual ASR with automatic TTS language switching
Omni Assistant	Omni model as a replacement for ASR + LLM with Magpie TTS
Omni Assistant Subagents	Multi-agent with live camera vision and attachment analysis
Frontend Backend Agent	Frontend conversation agent paired with a backend airline agent for tool calling

Agent Skills

Nemotron Voice Agent Builder Skill: Use the nemotron-voice-agent-builder skill to scaffold and refine real-time NVIDIA voice-agent pipelines, including Cascaded ASR -> LLM -> TTS and Nemotron Omni workflows, with Pipecat or LiveKit support for speech customization and cloud or local deployment.

Bash

npx skills add NVIDIA/skills --skill nemotron-voice-agent-builder

The skill empowers developers to rapidly build, customize, and deploy enterprise-grade voice agents for customer service and user interactions in healthcare, telecom, retail, and financial services.

Additional Voice Agent Examples

Nemo Voice Agent: A fully open-source and local evaluation pipeline for voice agents: GitHub Repository
NVIDIA XR AI: An open-source foundation for multimodal, real-time conversational AI in the NVIDIA CloudXR ecosystem. GitHub Repository
Daily/Pipecat - This repo is sample code for building voice agents with three NVIDIA open source models: Nemotron Speech ASR, Nemotron 3 Nano LLM, Magpie TTS (Preview) GitHub Repository. Read the blog post for more details. Deployed on DGX Spark.
Vertical examples - five examples from different verticals to converse with a voice agent - https://github.com/fciannella/nemotron-speech-demos/tree/main .
1. Claim Investigator (Tier 1)
2. Healthcare Agent
3. Banking Fee Inquiry Resolver
4. Telco Agent: Billing Specialist
5. Wire Transfer Agent with Twilio integration. (Call transfer) included.
Ambient Healthcare Agents + Guardrails - Github Repository

Ethical considerations

NVIDIA believes trustworthy AI is a shared responsibility. When using this example in accordance with our terms of service, work with your model and compliance teams to ensure the system meets requirements for your industry and use case. Report security or AI concerns here.