Tokenizer suggestion for fine tuning cache aware streaming model

[rkchamp25]

Hi
I want to fine tune "stt_en_fastconformer_hybrid_large_streaming_multi" on custom data.
In my dataset I have things like "Vitamin B12", "Code: c12r5", "hb1ac" etc
For these alphanumeric words:

1. Should I convert the above to "vitamin b twelve", "code c one two r five", "h b one a c" ..... for using the default tokenizer
2. Should I create a custom/new tokenizer for this?

If there is any other suggestion, please let me know.
Thank You

[titu1994]

If you want to finetune using the original tokenizer, yes you'll need to normalize all numbers to spoken words.

Changing tokenizer means you'll need a large amount of data to retrain the model, that is not suggested unless you have several thousand hours of speech to reach best results

[bfss]

If you want to finetune using the original tokenizer, yes you'll need to normalize all numbers to spoken words.

Changing tokenizer means you'll need a large amount of data to retrain the model, that is not suggested unless you have several thousand hours of speech to reach best results

How to use the original tokenizer?
I also created a discussion for this.