What does whisper_full_params.audio_ctx do?

[RndyP]

Comment says: /overwrite the audio context size (0 = use default)
Default is 1500. Suggestion by someone was to set to 768 for faster performance

What does this do?
What are the tradeoffs to setting to smaller value?

[ggerganov]

This is what I call to "partially evaluate the Encoder"

It is an idea suggested in #137 that makes the evaluation faster at the cost of accuracy. The original size of the audio context is 1500 (this corresponds to 30s audio), so you can set this to a smaller number. You always want the number to be multiple of 64 for better performance. For example, setting it to 768 would make the Encoder evaluate about 2 times faster, processing a ~15s audio chunk instead of a 30s audio chunk.

This option is useful when you are going to process short audio samples.
For example, it is suitable for the stream and command examples in this repo, when running on embedded devices or in the browser through WASM. It is one of the key ideas that allows to run the guided mode of the command on a Raspberry Pi:

command-guided-0.mp4

You would not want to use this option when you process long audio (i.e. more than 10-15s).
Some more info in #141

[RndyP]

For streaming then, would it make sense to adjust this value before each whisper_full(), based on the length of the chunk being passed?

[RndyP]

Shouldn't the default of 1500 be 1536?

[ggerganov]

You can try adjusting the value for streaming, but you don't want to make it too. I think anything below 768 is not good based on my experiments.

The default is 1500 because this is the original context size as defined by OpenAI.
We can pad the tensors in whisper.cpp with zeros to 1536 and this will probably result in some marginal performance improvement, but not sure if it is worth it - probably not.

[RndyP]

I used this to start at 768 and ramp up to 1536 at 30 seconds:
m_params.audio_ctx=min((64*(26*(int)m_pcmf32.size()/WHISPER_SAMPLE_RATE/64))+768,1536);

// run the inference
if (whisper_full(m_ctx,m_params,m_pcmf32.data(),(int)m_pcmf32.size())==0)