Overview

Speech Synthesis: Magpie TTS Multilingual Model Overview

Description:

The model is an end-to-end multilingual neural text-to-speech model that generates speech in twelve different languages (English-US, Spanish-US, German-German, French-France, Italian, Vietnamese, Mandarin-Chinese, Hindi, Japanese, Modern Standard Arabic, Brazilian Portuguese, and Korean) by predicting discrete audio codec tokens autoregressively using a transformer encoder-decoder architecture. It supports at least one male and one female speaker for all the languages. It employs multi-codebook prediction (typically 8 codebooks) with local transformer refinement and frame stacking (factor = 2) for high-quality audio generation, and leverages techniques like attention priors, classifier-free guidance (CFG), and Group Relative Policy Optimization (GRPO) for improved alignment and robustness. This release also includes a zero-shot inference fix, IPA grapheme-to-phoneme (G2P) support for custom dictionaries and code-switching, and updated G2P support for English-to-Katakana code-switching. The model supports both batch inference for complete utterances and long-form inference for very long text inputs with sliding window mechanisms. The generated codecs are then converted to a speech waveform using a frozen pretrained audio codec model.

This model is ready for commercial use.

License/Terms of Use:

GOVERNING TERMS: The NIM container is governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products; and the use of this model is governed by the NVIDIA Open Model License Agreement.

Deployment Geography:

Global

Use Case:

For streaming voice agent use-cases, For Offline speech generation from text, In multiple languages.

Release Date:

Build.Nvidia.com [06/22/2026] via [https://build.nvidia.com/nvidia/magpie-tts-multilingual]
NGC [06/22/2026] via [https://registry.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/magpie-tts-multilingual]

References(s):

TTS model papers:

• Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
• Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment

Audio codec paper:

• Low Frame-rate Speech Codec: A Codec Designed for Fast High-quality Speech LLM Training and Inference
  

Model Architecture:

Architecture Type: Transformer Encoder, Transformer Decoder, Local Transformer, and feedforward layers

Network Architecture:

1. Causal Transformer Encoder with 6 layers, learnable positional encoder of length 2048, and 1 Layer Normalization output layer.

2. Causal Transformer Decoder with 12 layers, learnable positional encoder of length 2048, and 1 Layer Normalization output layer.

3. Local Transformer for multi-codebook refinement, operating on stacked frames with a frame stacking factor of 2 to improve audio quality and reduce sequence length.
   

** Number of model parameters: 3.64 × 10^8 (364 M parameters)

Inference Configuration:

• Decoding: decoder-ce-context with GRPO-aligned weights
• Classifier-free guidance: cfg_scale = 2.5
• Sampling: temperature = 0.6, top_k = 80
• Attention prior: apply_attention_prior = True, prior_epsilon = 0.1, estimate_alignment_from_layers = [4, 5, 8, 9], apply_prior_to_layers = [2, 3, 4, 5, 6, 7, 8, 9, 10], start_prior_after_n_audio_steps = 0
  

Input(s):

Input Type(s): Text, Audio (optional reference utterance for zero-shot voice cloning)

Input Format(s):

• Text: Strings
• Audio: WAV
  

Input Parameters:

• Text: One-Dimensional (1D)
• Audio: One-Dimensional (1D)
  

Other Properties Related to Input: Text input is UTF-8 encoded; text normalization is required. Audio reference (when provided for voice cloning) is mono, PCM-encoded 16-bit WAV at a sampling rate of 22.05 kHz; recommended reference duration is between 3 and 10 seconds.

Output(s)

Output Type(s): Audio

Output Format(s):

• Audio: WAV
  

Output Parameters:

• Audio: One-Dimensional (1D)
  

Other Properties Related to Output: Mono, PCM-encoded 16 bit audio; sampling rate of 22.05 kHz; Audio output with dimensions (B x T), where B is batch size and T is time dimension.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• Riva 2.15.0
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
  • NVIDIA A100 GPU
  • NVIDIA A30 GPU
  • NVIDIA A10 GPU
    
• NVIDIA Hopper
  • NVIDIA H100 GPU
    
• NVIDIA Lovelace
  • NVIDIA L4 GPU
  • NVIDIA L40 GPU
    

Preferred/Supported Operating System(s):

• Linux
  
• Linux 4 Tegra
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

RivaTTS_MagpieTTS_Multilingual v4.0

Training and Evaluation Datasets:

Training Dataset:

The following datasets were used to train the model, including additional datasets focused on speech and ASR.

English (En):

• HiFiTTS En: link
  
• HiFiTTS-2 A Large-Scale High Bandwidth Speech Dataset En: link
  
• LibriTTS En: link
  
• GLOBE_V2 En (multi-accent): link
• DavidAI En: Internal Dataset
• Jensen Huang 2020 Keynote En: Internal Dataset
• RIVA Speakers En: Internal Dataset
• Publicly available internet-scale data En: Internal Dataset

German (De):

• CML-TTS De: Link
• King-ASR-182 De: Link

Spanish (Es):

• CML-TTS Es: Link
• ESP_ASR003 Es: Internal Dataset
• Riva Speakers Es: Internal Dataset

French (Fr):

• CML-TTS Fr: Link
• Riva Speakers Fr: Internal Dataset

Italian (It):

• CML-TTS It: Link
• MLS-Sidon It: Link

Hindi (Hi):

• AI4Bharat Kathbath Hi: link
• Hindi-English (MUCS) Hi: Link
• Publicly available internet-scale data Hi: Internal Dataset

Japanese (Ja):

• Common Voice Ja: Link
• Emilia YODAS Ja: Link
• Emilia YODAS Ja-En (code-switching): Link
• Japanese Anime Speech Ja: Link
• PlacesJapanese100k Ja: Link
• King-ASR-444 Ja: Link

Mandarin Chinese (Zh):

• AISHELL-3 Zh: Link
• THCHS-30 Zh: Link
• Emilia YODAS Zh: Link
• RIVA Speakers Zh: Internal Dataset
• Publicly available internet-scale data Zh: Internal Dataset

Vietnamese (Vi):

• InfoRe-1 Vi: Link
• Riva Speakers Vi: Internal Dataset
• Publicly available internet-scale data Vi: Internal Dataset

Brazilian Portuguese (pt-BR) — NEW:

• TTS-Portuguese-Corpus pt-BR: Link
• pt-br_char pt-BR: Link
• CML-TTS pt-BR: Link

Arabic — Modern Standard Arabic (ar_MSA) — NEW; also includes Saudi Arabia (ar_SA), UAE (ar_AE), and Damascene (ar_SY) accents:

• Common Voice Arabic: Link
• ArVoice Ar: Link
• YodaLingua Ar: Link
• Arabic Speech Corpus ar_SY: Link
• King-ASR-109 ar_AE: Link
• ClArTTS ar_SA: Link
• King-ASR-318 ar_SA: Link
• Publicly available internet-scale data Ar: Internal Dataset

Korean (Ko) — NEW:

• Emilia YODAS Ko: Link
• Zeroth-Korean Ko: Link

Data Modality

• [Audio]
  

Audio Training Data Size

• 54,305 Hours
  

Data Collection Method by dataset

• [Human]
  

Labeling Method by dataset

• [Hybrid: Human, Synthetic] - Human recorded data points were preprocessed algorithmically.
  

Properties: Number of data items in training set: ~54.3k hours Modality: Audio (speech signal) Nature of the content: Audio books, conversational speech, and read speech Language: Multilingual (En, Es, De, Fr, Vi, It, Zh, Hi, Ja, Ar, pt-BR, Ko) Sensor Type: Microphones

Evaluation Dataset:

• LibriTTS test-clean En: link
  
• CML-TTS Es: Link
  
• CML-TTS Fr: Link
  
• CML-TTS De: Link
  
• CML-TTS It: Link
  
• CML-TTS pt-BR: Link
  
• English accent test set (derived from GLOBE_V2): Link
  
• Internal Mandarin, Hindi, Japanese, Vietnamese, Arabic (MSA), and Korean test sets
  

Data Collection Method by dataset:

• [Human]
  

Labeling Method by dataset:

• [Human]
  
• [Hybrid: Human, Synthetic] - Human labeled data points are mixed and matched to create more variabilities.
  

Properties: Modality: Audio (speech signal) Nature of the content: Audio books, conversational speech, and newspaper passages Language: Multilingual (En, Es, De, Fr, Vi, It, Zh, Hi, Ja, Ar, pt-BR, Ko) Sensor Type: Microphones

Lower CER is better; higher SV-SSIM is better. MagpieTTS 4.0 (May 2026) shows CER and SV-SSIM improvements over MagpieTTS 3.1 across all previously supported languages, with the largest CER gains on Mandarin, French, and Italian. The English hard-set evaluation shows a substantial improvement on short prompts (CER reduced from 4.0% to 1.2%) while remaining comparable on Money, Digits, and Letters categories.

Inference:

Acceleration Engine: Triton
Test Hardware:

• NVIDIA A100 GPU
  
• NVIDIA A30 GPU
  
• NVIDIA A10 GPU
  
• NVIDIA H100 GPU
  
• NVIDIA L4 GPU
  
• NVIDIA L40 GPU
  

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.