Is it possible to disable the word normalization? #1883

[silvioprog]

Hi.

I have a couple of audios that the authors says: "gonna", "wanna", "let's", but I have it transcribed to "going to", "want to", "let us" etc.

Is it possible to disable this word normalization?

In the Python version, seems like it is possible here: https://github.com/openai/whisper/blob/ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab/whisper/normalizers/english.py#L465

Do we have something similar in the whisper.cpp?

TIA for any help!

[bobqianic]

Do we have something similar in the whisper.cpp?

AFAIK, whisper.cpp transcribes the text as it is, without any word normalization (as of February 21, 2024).

[silvioprog]

Yes I thougth the same, but please notice this transcription below:

% ./main --no-prints --language en --model ~/Downloads/ggml-large-v2-q5_0.bin --file ~/Downloads/audio.wav

In the transcribed result, we will see a going word at [00:08:51.520 --> 00:08:54.560] I get more excited about choosing what taco I'm going to get, but it should be gonna.

I have been using this web site to do the same transcription using the same model (large-v2), and it transcribed accordingly. For example, its result was: (8:51) I get more excited about choosing what taco I'm gonna get.

This is the link to download the audio. And the transcript from `whisper.cpp: whisper.cpp-transcript.txt.

I really would like to keep the words with their original accents without normalizations.

[misutoneko]

Just a quick observation...
I tested your sample with the original openai whisper (small model) and the --suppress_tokens 50364 trick gave me the "gonna" variation.
"I'm going to" is a hallucination that's sometimes seen with non-speech portions unless the --suppress_tokens trick is used.
EDIT: I didn't notice the same effect with "wanna" vs. "want to".
EDIT2: This seems to be model-specific...tried it with medium model and always got "I'm going to". Too bad.

[misutoneko]

It's the BEG token and yes, it can be suppressed, see #1592.
The model seems to be prone to generate the "I'm going to..." sequence whenever it sees the BEG token.
By eliminating this token we give the model the opportunity to consider other alternatives.

EDIT: Oh yeah I forgot, in whisper/whisper_timestamped the --suppress_tokens option also has a notable side-effect: It removes suppression of certain tokens.

[bobqianic]

It's the BEG token and yes, it can be suppressed, see #1592. The model seems to be prone to generate the "I'm going to..." sequence whenever it sees the BEG token. By eliminating this token we give the model the opportunity to consider other alternatives.

Why not simply use the no-timestamps mode instead of suppressing the BEG token?

[misutoneko]

Well, you could try that. I just haven't tried that approach. I kinda just randomly stumbled on this stuff and then used whatever works :D
And I'm not sure how much of it can be applied to whisper.cpp anyway.

[silvioprog]

@misutoneko I ran the ./main -h command and couldn't see any parameter to inform suppressed tokens. 😞

@bobqianic I've tried with no-timestamps, but the problem persisted. 😞

[misutoneko]

Yeah there isn't such parameter readily available, it requires a one line change in the source code and then recompilation.
But it looks as though this is not as effective in whisper.cpp as in the python variants, or at least I couldn't get it to handle the non-speech stuff on my previous try.
There is some work going on that may help, so maybe just wait a bit and retry after certain PRs have been merged :D
(hey, no pressure @bobqianic ;)

[misutoneko]

Looks like suppression doesn't help much, but what about prompting?
What I mean is, list all the possible variants in prompt.
(Just an idea, I haven't used prompting in this manner myself. But maybe?)

EDIT: Tested this idea a bit... yeah actually seems promising, but it only works with very short clips.
(This has been my experience in the past as well.)

I used these samples for testing (these are smaller sections cut out of the previous sample):
samples.zip