Non-English translations

[sanjaymk908]

Thanks @ggerganov for a phenomenal piece of s/w! OpenAI has def created fantastic models (more below). But my inference tests using the OpenAI Whisper impl on my puny MacBook Air and on beefy AWS servers were phenomenally bad :( 11 mins for a 30 sec audio clip e.g

Compare OpenAI Whisper w/ Whisper.cpp:
-Same puny MacBook Air
-Tiny model (77 Mb on disk)

• "-l 'auto'" coz my tests mostly involve non-english dialogs
• ~3-4 secs total for ~60 seconds audio
• Error rates < 10% when dialogs are in one lang mostly
• • Mixed dialogs (say english+Hindi) not handled well but not imp to me

But...have a Q re outputs with non-english audio. Why is it transliterated in english? e.g ""box office pe asa tandav kiya hai fund ne" is whisper.cpp's output. But its transliteration of "बॉक्स ऑफिस पर ऐसा तांडव किया है फंड ने"

Does this mean - OpenAI's models were trained on transliterated dialogs? One can use other APIs to change these back to Hindi e.g But the concern is - theres always some lossy conversions that happen. So, would be best to go from speech -> Hindi directly. Or am I missing something here?

[tinoue]

Did you add --language option ?

[ggerganov]

Also, make sure you use a multilingual model (i.e. one that does not end with .en, e.g. base.en, etc)

[sanjaymk908]

Thanks for the tips. Yes, I do use -l 'auto' & have tested w/ specific language settings (-l 'hindi' e.g). And I have been using the tiny model (no .en extension) converted via "bash ./models/download-ggml-model.sh tiny"

Observation: even the tiny model has staggeringly good transcribing quality on my personal tests that cover a lot of Indian languages.

[ggerganov]

Ah, I now read your question more carefully and understand the problem that you have.
I think it is possible that the model simply decides to use english transliteration instead of Hindi.

You should be able to fix this by passing a sample Hindi prompt to help the model "understand" what you want.
For example, try running this:

./main -m ./models/tiny.bin -l auto -p "यह हिंदी में कुछ यादृच्छिक पाठ है"

[sanjaymk908]

Thats a good tip, thanks! I should experiment w/ prompts.
I also got the ggml-medium.bin & I can see "./main -m ./models/ggml-medium.bin -l auto" detecting languages much better. And also producing transcripts in the desired language now.
It has expected side-effects however - larger models consuming more mem+time. So, definitely needs beefier Macs.

[thewh1teagle]

I'm also trying to use translate from english to Hebrew but it always transcribe in English.