Further improve the speed of Whisper.cpp (might need a dependency)

[haikyuu]

I came across Faster Whisper which is 5x faster than whisper.cpp with comparable memory footprint. It's built in python and uses this C++ library (Ctranslate2)

Just bringing this to your attention as this is very compelling ... 5x faster whisper.cpp (plus models are 4x smaller on disk) would unlock so much more use cases

[guillaumekln]

Hey, thanks for sharing this here! I'm the author of faster-whisper and CTranslate2.

I was a bit surprised by the benchmark results on this CPU. I attribute most of the gains to our integration of Intel oneMKL which basically offers unmatched performance for matrix multiplication. Also oneMKL can make use of AVX512 instructions, which are supported by this processor.

I just relaunched the whisper.cpp fp16 run to make sure I did not make a mistake. For reference, below is the command that I used and the output logs (with commit 3b010f9):

$ /usr/bin/time -v ./main -t 8 -bs 5 -l auto -m models/ggml-small.bin -f output.wav
whisper_init_from_file: loading model from 'models/ggml-small.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51865
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 768
whisper_model_load: n_audio_head  = 12
whisper_model_load: n_audio_layer = 12
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 768
whisper_model_load: n_text_head   = 12
whisper_model_load: n_text_layer  = 12
whisper_model_load: n_mels        = 80
whisper_model_load: f16           = 1
whisper_model_load: type          = 3
whisper_model_load: mem required  =  603.00 MB (+   16.00 MB per decoder)
whisper_model_load: kv self size  =   15.75 MB
whisper_model_load: kv cross size =   52.73 MB
whisper_model_load: adding 1608 extra tokens
whisper_model_load: model ctx     =  464.56 MB
whisper_model_load: model size    =  464.44 MB

system_info: n_threads = 8 / 15 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

main: processing 'output.wav' (12786951 samples, 799.2 sec), 8 threads, 1 processors, lang = auto, task = transcribe, timestamps = 1 ...

whisper_full: auto-detected language: fr (p = 0.987908)

[...]

whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:     load time =   647.20 ms
whisper_print_timings:      mel time =  3202.05 ms
whisper_print_timings:   sample time = 166569.81 ms / 18858 runs (    8.83 ms per run)
whisper_print_timings:   encode time = 131752.80 ms /    32 runs ( 4117.27 ms per run)
whisper_print_timings:   decode time = 437354.44 ms / 18735 runs (   23.34 ms per run)
whisper_print_timings:    total time = 740297.12 ms
        Command being timed: "./main -t 8 -bs 5 -l auto -m models/ggml-small.bin -f output.wav"
        User time (seconds): 4618.27
        System time (seconds): 20.27
        Percent of CPU this job got: 626%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 12:20.37
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 873572
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 336643
        Voluntary context switches: 18972
        Involuntary context switches: 57161
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Let me know if you have any questions. I would be happy to help making whisper.cpp faster!

[guillaumekln]

Hello,

Indeed, when using beam_size = 1 there is big time reduction with whisper.cpp, much more than with faster-whisper:

Implementation	Precision	Beam size	Time	Max. memory
whisper.cpp	fp16	1	4m03s	724MB
faster-whisper	fp32	1	1m42s	1893MB
faster-whisper	int8	1	1m17s	873MB

whisper.cpp command and logs for reference

/usr/bin/time -v ./main -t 8 -bs 1 -bo 1 -l auto -m models/ggml-small.bin -f output.wav
whisper_init_from_file: loading model from 'models/ggml-small.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51865
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 768
whisper_model_load: n_audio_head  = 12
whisper_model_load: n_audio_layer = 12
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 768
whisper_model_load: n_text_head   = 12
whisper_model_load: n_text_layer  = 12
whisper_model_load: n_mels        = 80
whisper_model_load: f16           = 1
whisper_model_load: type          = 3
whisper_model_load: mem required  =  603.00 MB (+   16.00 MB per decoder)
whisper_model_load: kv self size  =   15.75 MB
whisper_model_load: kv cross size =   52.73 MB
whisper_model_load: adding 1608 extra tokens
whisper_model_load: model ctx     =  464.56 MB
whisper_model_load: model size    =  464.44 MB

system_info: n_threads = 8 / 15 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

main: processing 'output.wav' (12786951 samples, 799.2 sec), 8 threads, 1 processors, lang = auto, task = transcribe, timestamps = 1 ...

whisper_full: auto-detected language: fr (p = 0.987908)

[...]

whisper_print_timings:     fallbacks =   2 p /   0 h
whisper_print_timings:     load time =   599.06 ms
whisper_print_timings:      mel time =  3522.00 ms
whisper_print_timings:   sample time =  2811.51 ms /  3723 runs (    0.76 ms per run)
whisper_print_timings:   encode time = 137409.83 ms /    33 runs ( 4163.93 ms per run)
whisper_print_timings:   decode time = 99429.89 ms /  3724 runs (   26.70 ms per run)
whisper_print_timings:    total time = 243845.12 ms
        Command being timed: "./main -t 8 -bs 1 -bo 1 -l auto -m models/ggml-small.bin -f output.wav"
        User time (seconds): 1897.34
        System time (seconds): 5.42
        Percent of CPU this job got: 780%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 4:03.88
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 724412
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 209111
        Voluntary context switches: 3874
        Involuntary context switches: 16399
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

I confirm that for both beam_size and best_of you could batch the different entries in a single input. This adds a new batch dimension so this may require changes in your implementation.

[ggerganov]

Thanks!

Do you have some estimates of how oneMKL GEMM compares to OpenBLAS GEMM?
In my experience, I observed that when I use a FP32 model with OpenBLAS GEMM I get poorer performance compared to FP16 model with AVX + F16C SIMD. This was a bit surprising to me, because I am sure my matrix multiplication implementation is very far from optimal, but still it outperformed the OpenBLAS F32 one. I attributed the difference to hitting a memory bandwidth limit.
But, now I see that your implementation is significantly faster with FP32 and I am starting to doubt my conclusions. Although the AVX512 instruction set might explain things. Not sure.

I see that CTranslate2 supports Apple Accelerate. Do you have performance benchmarks on Apple Silicon?
It would be interesting to compare since in this environment, whisper.cpp also uses Apple's BLAS and so the matrix multiplication will be the same. If there is still difference, then it would mean there is some other performance hit.

I'll definitely run some comparisons with your code when I get some time and will report results here.

[guillaumekln]

Unfortunately I don't have access to an Apple machine at the moment. However, we do support OpenBLAS. I will run a more extensive comparison with oneMKL vs OpenBLAS and AVX2 vs AVX512.

[ggerganov]

One suggestion - I think it would make more sense to compare the Encoder time per run and Decoder time per run since due to the auto-regressive nature of the model the 2 code bases could branch into different transcription results and effectively perform different number of inferences for the same audio input. In the end, there is no optimal decoding strategy, so one is mostly interested in having an optimal inference of the model per run.

[guillaumekln]

FYI, here are additional data points comparing AVX2/AVX512 and oneMKL/OpenBLAS for a fixed code base.

It seems that AVX512 does not have a big impact, at least for the small model and the CPU used in this test. However, oneMKL is noticeably faster than OpenBLAS.

Implementation	Backend	ISA	Precision	Beam size	Time for 13m of audio
faster-whisper	OpenBLAS	AVX2	fp32	1	3m06s
faster-whisper	OpenBLAS	AVX512	fp32	1	2m54s
faster-whisper	oneMKL	AVX	fp32	1	2m24s
faster-whisper	oneMKL	AVX2	fp32	1	1m50s
faster-whisper	oneMKL	AVX512	fp32	1	1m50s
faster-whisper	oneMKL	AVX2	int8	1	1m25s
faster-whisper	oneMKL	AVX512	int8	1	1m17s