# Speech Recognition/Translation: Canary ## Description RIVA Canary-1B (around 1B parameters) [1] is a multi-lingual multi-tasking model that supports automatic speech-to-text recognition (ASR) and automatic speech-to-text translation (AST) in following languages: Arabic (ar-AR), English (en-US, en-GB), Spanish (es-US, es-ES), German (de-DE), French (fr-FR), Hindi (hi-IN), Italian (it-IT), Portuguese (pt-BR), Japanese (ja-JP), Korean (ko-KR), and Russian (ru-RU) with punctuation and capitalization (PnC). Mandarin (zh-CN) is supported as a target language in the translation task. ## License/Terms of Use [NVIDIA AI Foundation Models Community License Agreement](https://docs.nvidia.com/ai-foundation-models-community-license.pdf) ## References [1] [Less is More: Accurate Speech Recognition & Translation without Web-Scale Data](https://arxiv.org/pdf/2406.19674)
[2] [New Standard for Speech Recognition and Translation from the NVIDIA NeMo Canary Model](https://developer.nvidia.com/blog/new-standard-for-speech-recognition-and-translation-from-the-nvidia-nemo-canary-model/)
[3] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
[4] [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
## Model Architecture **Architecture Type:** An encoder-decoder model with FastConformer [3] encoder and Transformer decoder [4]
**Network Architecture:** 24 layer encoder, 24 layer decoder
## Input **Input Type(s):** Audio
**Input Format(s):** wav
**Input Parameters:** 1-Dimension
**Other Properties Related to Input:** Mono channel is required
## Output **Output Type(s):** Text
**Output Format:** String
**Output Parameters:** 1-Dimension
**Other Properties Related to Output:** No Maximum Character Length, Does not handle special characters
## How to Use this Model The [Riva Quick Start Guide](https://ngc.nvidia.com/catalog/resources/nvidia:riva:riva_quickstart) is recommended as the starting point for trying out Riva models. For more information on using this model with Riva Speech Services, see the [Riva User Guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-overview.html). ## Suggested Reading Refer to the [Riva documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/index.html) for more information. ## Software Integration **Runtime Engine(s):** * Riva 2.18.0 or higher
**Supported Hardware Microarchitecture Compatibility:**
* NVIDIA Ampere
* NVIDIA Hopper
* NVIDIA Jetson
* NVIDIA Turing
* NVIDIA Volta
**[Preferred/Supported] Operating System(s):**
* Linux
* Linux 4 Tegra
## Model Version(s): Canary-1b 1.0 # Training & Evaluation ## Training Dataset ** Data Collection Method by dataset
* Human
** Labeling Method by dataset
* ASR: Human
* AST: Human, Synthetic
**Properties:**
Mixture of organic ASR data aligned with human voices and machine generated translations to create AST pairings. ## Evaluation Dataset ** Data Collection Method by dataset
* Human
** Labeling Method by dataset
* Hybrid: Human, Synthetic
**Properties:** A dynamic blend of public and internal proprietary and customer datasets aligning text with human audio data. ## Inference **Engine:** Triton
**Test Hardware:**
* NVIDIA A10
* NVIDIA A100
* NVIDIA A30
* NVIDIA H100
* NVIDIA L4
* NVIDIA L40
* NVIDIA Turing T4
* NVIDIA Volta V100
## Ethical Considerations (For NVIDIA Models Only): NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).