Adding own "dictionary" or phrase pairs

[mirozahorak]

First of all, amazing work @ggerganov & the rest!
I wonder if the models include translating from AI vocabulary :) I am programmer myself, but when i read your discussion it is like a foreign language to me. Just kidding, no need to implement :)

To my point: Is there a chance of adding my own phrase pairs or word pairs so that transcript precision for certain specific topics can be enhanced? Like names or phrases from specialised areas. From my tests I have noticed that this is area of most inaccurate transcripts.

The same applies to translating, but that is more complex topic.

[RndyP]

gerganov has the command example that does this. I'm in the same boat as you. You can create a custom dictionary. I had it up and running. The problem was false positives. When selecting from a dictionary it will choose the highest probability phrase. The problem is that the probability value is all over the place sometimes, and I couldn't depend on it. gerganov has this problem on his radar, but is swamped with other things. I can't find the issue # right now.

Anyway, what I did to get my preferred words to work is create an error dictionary. I "train" it by saying the word over and over and adding all the incorrect words to the dictionary.

[mirozahorak]

Thanks for the insight. Error dictionary solution is something I have not heard about, neither I knew that i can actually train the model myself.
I need to search more in the repo for this example.

To avoid false positives – maybe there is a way how to increase the probability by percentage, relatively?

[RndyP]

I think what you are looking for is in between regular inference, and a special "command" dictionary. You want to favor certain words. I'm thinking to do that, you might want a custom ggml*.bin file. Whisper is a weakly trained, large dataset, that has every word under the sun in it. Would be nice if there could be filtering, like, no names adjectives, scrabble words, etc.....