Short audio files not decoded (empty output)

[jploski]

Reporting against a750868 with ggml-large-v3-q5_0.bin. If the file to decode is very short (e.g. < 1s, single recorded word), the GGML version will consistently fail to output anything at all:

whisper.cpp/bin/main -nt -m /mnt/f2fs/whisper/ggml-large-v3-q5_0.bin -f /tmp/sample0000.wav
...
main: processing '/tmp/sample0001.wav' (8879 samples, 0.6 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 0 ...

(no text output)

But if the same file is padded with 1 second of silence in front, it starts working:

sox /tmp/sample0000.wav /tmp/sample0001.wav pad 1@0
whisper.cpp/bin/main -nt -m /mnt/f2fs/whisper/ggml-large-v3-q5_0.bin -f /tmp/sample0001.wav
...
main: processing '/tmp/sample0001.wav' (24879 samples, 1.6 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 0 ...


 Down.

It's an easy enough workaround, but why doesn't it just work unpadded? I don't have the same issue with the whisper-large-v3 in Python.

[ggerganov]

There is a check to stop processing if less than 1s of audio remains:

whisper.cpp/whisper.cpp

Lines 5271 to 5277 in a750868

	// if length of spectrogram is less than 1.0s (100 frames), then return
	// basically don't process anything that is less than 1.0s
	// see issue #39: https://github.com/ggerganov/whisper.cpp/issues/39
	if (seek_end < seek_start + (params.speed_up ? 50 : 100)) {
	WHISPER_LOG_DEBUG("%s: input is too short - %d ms < 1000 ms\n", __func__, (seek_end - seek_start)*10);
	return 0;
	}

I've figured it helps in most situations, but obviously can affect negatively in cases like yours. This should become an input parameter

[jploski]

There is a check to stop processing if less than 1s of audio remains:

whisper.cpp/whisper.cpp

Lines 5271 to 5277 in a750868

	// if length of spectrogram is less than 1.0s (100 frames), then return
	// basically don't process anything that is less than 1.0s
	// see issue #39: https://github.com/ggerganov/whisper.cpp/issues/39
	if (seek_end < seek_start + (params.speed_up ? 50 : 100)) {
	WHISPER_LOG_DEBUG("%s: input is too short - %d ms < 1000 ms\n", __func__, (seek_end - seek_start)*10);
	return 0;
	}

I've figured it helps in most situations, but obviously can affect negatively in cases like yours. This should become an input parameter

Thanks for pointing it out. However, even if I comment out this block of code, the result is the same (no output).

If it improves behavior in some scenarios, the guard condition could stay as is, but it would be nice if a more prominent output message was shown at least when running via "main" in interactive mode (otherwise it just looks as if it's "broken somehow", which may be off-putting for first-time users).

(And quick processing of short audio files is needed for implementing interactive voice interfaces a'la Talon/Alexa etc.)

[royshil]

@jploski you can pad your input with 0s until you hit 1000ms in length

[itsthisjustin]

I'm not 100% sure if this is the same issue, but I'm currently having a problem where if the first bit of audio is blank it just doesnt bother transcribing the rest. So if my audio file has say 10 seconds of silence at the beginning but then a minute of audio after that, it outputs a blank transcript.

I think what's happening is if the first segment is empty, whisper doesn't try for the next segment.

(Small update, I think this only occurs if you have single segment set to true)