NeMo seems to freeze when I try to decode with a LM using the script eval_beamsearch_ngram_ctc.py

[rjimersonjr]

I trained a model using this tutorial. I'm trying to decode using that model with an LM. I installed the decoders, but the applications seem to hang during the decoding. I'm using the eval_beamsearch_ngram_ctc.py script.

I am seeing an interesting warning...

[NeMo W 2024-02-01 22:34:31 nemo_logging:349] /home/username/anaconda3/envs/nemo/lib/python3.10/site-packages/numba/cuda/
cudadrv/nvvm.py:366: UserWarning: CUDA Toolkit 10.2 is unsupported by Numba - 11.2 is the minimum required version.          
      warnings.warn(unsupported_ver)

I followed the instructions on the github page, and it installs Python-Cuda 11.8 in the Conda environment. The Cuda version on the server is CUDA 12.0. So I'm not sure where the 10.2 came from. It does say that it may take some time, but I've let it run for a couple days and the script still hasn't finished. Below is the output set to the screen.

20240130_run_ctc.txt

[michiahe]

@rjimersonjr

I discovered your post yesterday and I think that I have the same issue as you have. Since there have been 2 months since your post, have you found a solution to your issue?

In my case, the issue showed up after updating from a very old NeMo version to 1.22 and Nvidia Pytorch 23.05 or 23.12 Docker containers. I was able to confirm that changing the import of ctc_decoders from inside __init__ to a module level import resolved the issue. I.e. move the following line to the top of the module.

NeMo/nemo/collections/asr/modules/beam_search_decoder.py

Lines 63 to 69 in 1b0c6c9

	try:
	from ctc_decoders import Scorer, ctc_beam_search_decoder_batch
	except ModuleNotFoundError:
	raise ModuleNotFoundError(
	"BeamSearchDecoderWithLM requires the installation of ctc_decoders "
	"from scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh"
	)

Digging deeper into the problem, I figured that it might be a deadlock caused when importlib tries to import the contents of _swig_decoders (the internal c++ implementation of swig_decoders used inside ctc_decoders). I don't know any final solution for the issue and would appreciate if anyone else could comment on it.

[titu1994]

We're moving to Flashlight as the defacto engine for CTC Beam Search and will be deprecating OS2S beam decoder in the coming release due to its unmaintained status

Fyi @karpnv

[michiahe]

I see. Thanks for the reply and I can understand the decision since OS2S decoders aren't maintained anymore.

This isn't good news for us, though because we had troubles with Flashlight decoders and Japanese CTC ASR models trained on the CSJ. Flashlight had worse CER than greedy decoding. Let me see if I can reproduce these results on another model and data combination (I'm thinking of the models released by Reazon #8473).

[titu1994]

This is interesting, please let us know what all youve tried to make flashlight work on your case. We assumed it was a drop in replacement for OS2S.

We do have a Japanese Fast Conformer model released by an external team, we might be able to see if we can or cannot improve cer on japanese with subword models - https://huggingface.co/reazon-research/reazonspeech-nemo-v2

Their original post - #8473

[michiahe]

I think I figured out the issue. There is a bug in NeMo's implementation for lexicon-free decoding with Flashlight. The fix is just 2 lines but let me test it. I'll make another PR or add to my current PR.

[michiahe]

I checked with our internal test sets and my bugfix works. I get very similar CERs to the OS2S decoders. Let me add it to my current PR (I'll rename it) #8856