Beam search too slow on the transducer-conformer recipe

[asumagic]

Describe the bug

Evaluating LibriSpeech test-clean can take >1h instead of something like two minutes.

I am not sure if this affects other models. Greedy search is much faster in comparison.

We've already brought this up internally a few times but I realized we didn't have a formal issue to link to for it, so here it is.

Expected behaviour

It should not take as long. Unsure where exactly the fault lies (TransducerBeamSearch or simply the parameters?), I didn't diagnose this much yet. Profiling did not show anything peculiar other than that evaluating the LSTM took roughly a third~half of that time and that there was a high number of iterations.

To Reproduce

The following recipe is affected during evaluation: https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriSpeech/ASR/transducer

If required I can try to provide a python script repro.

Environment Details

No response

Relevant Log Output

No response

Additional Context

No response

[TParcollet]

It's not slow ... it is slow AS FUCK. But I believe that this is expected from python only RNN6T decoders.