Timestamps with Canary transcripts

[GreatHalf]

Hello,
it appears that all compute_timestamps options are ignored when using Canary, e.g. with examples/asr/speech_multitask/speech_to_text_aed_chunked_infer.py.

Furthermore, you cannot use preserve alignments, although these might be used to help Nemo Forced Aligner (as I stated here). The warning when setting this flag is [NeMo I 2024-04-12 15:46:02 multitask_beam_decoding:159] Preservation of alignments was requested but TransformerAEDBeamInfer does not implement it.

As far as I can tell, the only way to get Canary results with timestamps is:

1. transcribe with Canary to get the text
2. run NFA with the text (and recalculate fitting alignments)

Is this correct?
Is this due to the aed approach used by Canary? Can we expect timestamp support in the future? Whisper eventually supported them.

Cheers

[GreatHalf]

I found the answer here https://huggingface.co/nvidia/canary-1b/discussions/16. Everything appears to be as I suspected.

I would like to keep this open in case of new development that happened in the meantime.

[titu1994]

NFA combining Canary Transcripts with Parakeet CTC aligner is going to give much more accurate alignments than using greedy alignment algorithm that we have for CTC RNNT. We will add some documentation for it. Fyi @erastorgueva-nv

[GreatHalf]

Parakeet only has English models available, right?
So it's not an out-of-the-box option for the non-English transcripts of Canary.

[titu1994]

You can use any CTC or hybrid model for alignment, we have multiple models for each language that canary supports