DEV Community

Daniel Guerrero
Daniel Guerrero

Posted on

Nvidia NeMo Speech Recognition Starting Guide

#asr #nemo #ai

After reading an article that nvidia had several models in top of HF ASR Leaderboard I wanted to test in my local computer.
Even the code seems pretty simple from HF it turns didn't work for nvidia/canary-qwen-2.5b so I started to dig a bit more and test several features.

Base Setup

To test you will need a base setup, I'm using docker so the options are:

  1. Using Nvidia PyTorch container
  2. Using python with cuda enabled libraries

Nvidia PyTorch container

This is the simplest, but of course could contain a lot of libraries not needed, the size of the container is 12.78GB

docker run \ --gpus all \ -it \ --rm \ --shm-size=16g \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ nvcr.io/nvidia/pytorch:25.06-py3
Enter fullscreen mode Exit fullscreen mode

Python image with cuda libraries 

docker run \ --gpus all \ -it \ --rm \ python:3.12-bookworm \ /bin/bash
Enter fullscreen mode Exit fullscreen mode

Setup cuda libraries:

apt update && \ apt install -y wget && \ wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb && \ dpkg -i cuda-keyring_1.1-1_all.deb && \ apt-get update && \ apt-get -y install --no-install-recommends cuda-toolkit-12-9
Enter fullscreen mode Exit fullscreen mode

Note: the image is around 11.5Gb so probably is not really much different from Nvidia container image.

Setup NeMo libraries

This is pretty simple:

pip install nemo-toolkit[asr]
Enter fullscreen mode Exit fullscreen mode

Setup ffmpeg

For the examples should not be needed but libraries will report missing and is good to have for creating proper input files

cd /tmp wget https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz tar xvf ffmpeg-master-latest-linux64-gpl.tar.xz cp ffmpeg-master-latest-linux64-gpl/bin/* /usr/bin/ rm -rf ffmpeg-master-latest-linux64-gpl ffmpeg-master-latest-linux64-gpl.tar.xz
Enter fullscreen mode Exit fullscreen mode

Get audio samples

You need audio samples that are:

  • 1 single channel
  • 16Khz Sample Rate
  • less than 20 seconds

An example used on several examples is this:
https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav

so you can download:

wget https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav
Enter fullscreen mode Exit fullscreen mode

This sample have the previous requirements, so is not needed to process; but if you want to have a proper sample from any file you can do with ffmpeg:

ffmpeg -i INPUT_FILE -ac 1 -ar 16000 example.wav
Enter fullscreen mode Exit fullscreen mode

A more complex can be to extract just a part of the audio, the following will get 10 seconds, starting at second 29 so will extract audio from 29s - 39s:

ffmpeg -i INPUT_FILE -s 29 -t 10 -ac 1 -ar 16000 example.wav
Enter fullscreen mode Exit fullscreen mode

Code example

The following file (asr_example.py) will help you to test the different models:

import nemo.collections.asr as nemo_asr import argparse parser = argparse.ArgumentParser(prog="ASR NeMo Example") parser.add_argument( "--enable-timestamps", help="Enable timestamps", action=argparse.BooleanOptionalAction, ) parser.add_argument( "model_name", help="Name of the model like 'nvidia/canary-1b-flash'", ) parser.add_argument( "input_file", help="Path of the wav file, must be 16000Hz and 1 channel", ) args = parser.parse_args() asr_model = nemo_asr.models.ASRModel.from_pretrained(args.model_name) transcriptions = asr_model.transcribe( args.input_file, timestamps=args.enable_timestamps, ) for idx, transcript in enumerate(transcriptions): print(f"[{idx}] {transcript.text}") if args.enable_timestamps: for stamp in transcript.timestamp["word"]: word = stamp['word'] output_line = f"{stamp['start']:0>5.2f}" output_line += f"-{stamp['end']:0>5.2f}" output_line += f": {word}" print(output_line)
Enter fullscreen mode Exit fullscreen mode

Here is a list of models: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/models.html
Note that list is not fully updated as the most recent is nvidia/canary-qwen-2.5b but this won't work with current code.

Testing the code

You need to provide the model name and the input file, so you can call like this:

python3 asr_example.py \ nvidia/canary-1b-flash \ 2086-149220-0033.wav
Enter fullscreen mode Exit fullscreen mode

This will output:

[0] Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait.
Enter fullscreen mode Exit fullscreen mode

You can enable the timestamps per word (not all the models support timestamps, but it will trigger an error if doesn't support it)

python3 asr_example.py \ --enable-timestamps \ nvidia/canary-1b-flash \ 2086-149220-0033.wav
Enter fullscreen mode Exit fullscreen mode

This will output:

[0] Well I don't wish to see it any more observed Phoebe turning away her eyes it is certainly very like the old portrait 00.32-00.40: Well 00.56-00.72: I 00.72-01.04: don't 01.04-01.28: wish 01.28-01.36: to 01.44-01.52: see 01.60-01.68: it 01.76-01.84: any 01.92-02.00: more 02.24-02.64: observed 02.64-03.12: Phoebe 03.36-03.68: turning 03.76-03.84: away 04.08-04.16: her 04.24-04.48: eyes 04.96-05.04: it 05.12-05.20: is 05.36-05.76: certainly 05.84-05.92: very 06.08-06.16: like 06.24-06.32: the 06.40-06.48: old 06.64-07.12: portrait
Enter fullscreen mode Exit fullscreen mode

Multilanguage

One of the cool features of the Canary family is the support of multiple input languages (English, German, French, Spanish) and can even translate the output.
I will use one file from this dataset: https://www.kaggle.com/datasets/carlfm01/120h-spanish-speech

In the case of the language instead of passing the wav file, it needs to be created an input manifest json.
The format is like this:

{ "audio_filepath": "FILE.wav", "duration": 10, "source_lang": "es", "target_lang": "en" }
Enter fullscreen mode Exit fullscreen mode

But the trick is the input file is actually a text file where each line is a json entry, so the input-spanish.json must be:

{"audio_filepath": "0000df16-47ea-428f-8367-df2ce365d5c4.wav","duration": 9, "source_lang": "es","target_lang": "es"}
Enter fullscreen mode Exit fullscreen mode

And to run with:

python3 asr_example.py \ --enable-timestamps \ nvidia/canary-1b-flash \ input-spanish.json
Enter fullscreen mode Exit fullscreen mode

The output will be:

[0] con efeto, su lenguaje y singulares maneras me divertían extraordinariamente, porque nuestro hombre era un verdadero andaluz, 00.00-00.08: con 00.48-01.04: efeto, 01.12-01.20: su 01.36-01.84: lenguaje 01.92-02.00: y 02.08-02.64: singulares 02.72-03.04: maneras 03.20-03.28: me 03.36-03.92: divertían 04.08-05.92: extraordinariamente, 05.92-06.00: porque 06.40-06.48: nuestro 06.88-06.96: hombre 07.28-07.36: era 07.52-07.60: un 07.68-08.16: verdadero 08.24-08.96: andaluz,
Enter fullscreen mode Exit fullscreen mode

And if you want to translate into english, the input-spanish.json must be:

{"audio_filepath": "0000df16-47ea-428f-8367-df2ce365d5c4.wav","duration": 9, "source_lang": "es","target_lang": "en"}
Enter fullscreen mode Exit fullscreen mode

In this case the output of the same command will be:

[0] with effect his language and singular manners amused me extraordinarily because our man was a true Andalusian 00.00-00.08: with 00.48-00.56: effect 01.12-01.20: his 01.36-01.76: language 01.84-01.92: and 02.00-02.56: singular 02.64-03.04: manners 03.20-03.84: amused 04.08-04.16: me 04.24-05.28: extraordinarily 05.92-06.00: because 06.40-06.48: our 06.80-06.88: man 07.20-07.28: was 07.44-07.52: a 07.60-07.68: true 08.16-08.80: Andalusian
Enter fullscreen mode Exit fullscreen mode

Top comments (0)