Speech Recognition/Translation: Canary

Description

RIVA Canary-1B (around 1B parameters) [1] is a multi-lingual multi-tasking model that supports automatic speech-to-text recognition (ASR) and automatic speech-to-text translation (AST) in following languages: Arabic (ar-AR), English (en-US, en-GB), Spanish (es-US, es-ES), German (de-DE), French (fr-FR), Hindi (hi-IN), Italian (it-IT), Portuguese (pt-BR), Japanese (ja-JP), Korean (ko-KR), and Russian (ru-RU) with punctuation and capitalization (PnC). Mandarin (zh-CN) is supported as a target language in the translation task.

License/Terms of Use

NVIDIA AI Foundation Models Community License Agreement

References

[1] Less is More: Accurate Speech Recognition & Translation without Web-Scale Data
[2] New Standard for Speech Recognition and Translation from the NVIDIA NeMo Canary Model
[3] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
[4] Attention Is All You Need

Model Architecture

Architecture Type: An encoder-decoder model with FastConformer [3] encoder and Transformer decoder [4]
Network Architecture: 24 layer encoder, 24 layer decoder

Input

Input Type(s): Audio
Input Format(s): wav
Input Parameters: 1-Dimension
Other Properties Related to Input: Mono channel is required

Output

Output Type(s): Text
Output Format: String
Output Parameters: 1-Dimension
Other Properties Related to Output: No Maximum Character Length, Does not handle special characters

How to Use this Model

The Riva Quick Start Guide is recommended as the starting point for trying out Riva models. For more information on using this model with Riva Speech Services, see the Riva User Guide.

Suggested Reading

Refer to the Riva documentation for more information.

Software Integration

Runtime Engine(s):

• Riva 2.18.0 or higher
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
  
• NVIDIA Hopper
  
• NVIDIA Jetson
  
• NVIDIA Turing
  
• NVIDIA Volta
  

[Preferred/Supported] Operating System(s):

• Linux
  
• Linux 4 Tegra
  

Model Version(s):

Canary-1b 1.0

Training & Evaluation

Training Dataset

** Data Collection Method by dataset

• Human
  

** Labeling Method by dataset

• ASR: Human
  
• AST: Human, Synthetic
  

Properties:

Mixture of organic ASR data aligned with human voices and machine generated translations to create AST pairings.

Evaluation Dataset

** Data Collection Method by dataset

• Human
  

** Labeling Method by dataset

• Hybrid: Human, Synthetic
  

Properties:

A dynamic blend of public and internal proprietary and customer datasets aligning text with human audio data.

Inference

Engine: Triton
Test Hardware:

• NVIDIA A10
  
• NVIDIA A100
  
• NVIDIA A30
  
• NVIDIA H100
  
• NVIDIA L4
  
• NVIDIA L40
  
• NVIDIA Turing T4
  
• NVIDIA Volta V100
  

Ethical Considerations (For NVIDIA Models Only):

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns here.