Benchmark results #89
Comments
|
Results for
|
|
Results for Ryzen 5 4500U 6C/6T laptop CPU (I've just included one result for 8 threads as Encode time is much higher when threads > CPU cores).
|
|
|
|
|
This performance is impressing!
|
Yes, there is a huge performance boost due to using the built-in BLAS implementation on these devices. I will soon add OpenBLAS support for x86 architectures and see how this compares. By the way, AVX-512 is not supported on |
compiled with MinGW64 gcc 11.3 |
|
Valve Jupiter (AMD Custom APU 0405, Zen 2 microarch, 4c8t, 16GB DDR5 @ 5200 MT/s)
Compiled with The performance gains on jfk.wav since last test (two weeks or so ago) are extremely impressive, ~10-20x speedup from 40 to 2-4 seconds. |
|
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
@trholding You can generate a table with performance results by simply running the extra/bench_all.sh script. Regarding the threads - yes, it seems that going beyond 8 threads does not help regardless of how many cores you have. My guess is that the computation is memory-bound so that's why using more threads does not improve the performance. |
This comment was marked as outdated.
This comment was marked as outdated.
Hey Sorry. That didn't pan out well, I did the benchmark thrice, my account got deleted without notice. Could't get the logs as it was a web terminal. On the other hand I am happy that this happened, I was giving serious thought of purchasing a GPU+CPU plan there, so performance check of CPU was equally important. Probably or technically it was my fault - probably shouldn't have used a reverse shell and done benchmarks on a free trial, but how does one know if a service is real good or all just vapor... |
|
Dell Precision 5560 laptop results:
|
|
|
Compiled with VS 2022 Something is off, right? |
|
Yup - you are missing the |
|
OK, the
Compiled with VS 2022 |
|
|
From the stream repo
I still haven't worked out the little(0-3).Big(4-7) on this thing as if I pin to big cores
I tried to compile with openBlas but seemed to kill the make From the master repo as didn't think about the repo after trying streaming input
|
8 threads seemed to be the fastest. However I managed to squeeze a bit more performance by pinning CPU:
|
|
Results for AWS Graviton 3 Processor ( Compiled with
|
|
@matth Do you observe significant performance difference with / without |
|
@ggerganov
Results without any
I have tried to improve by using OpenBlas and Are there any possibilities for further optimisations in |
ThinkPad P1 Gen 4Intel Core i7-11800H | 64GB RAM | NVIDIA T1200 4GB GPU Encoder
memcpyggml_mul_mat |
|
Me and my friends did some benchmark and profiling. Apparently llama.cpp only uses the GPU for matrix multiplication. And currently the major time on CPU being Conv1D and softmax. So that'll be a good starting point to improve the performance on CUDA/OpenCL. |
Jetson Nano 4GB using cuBLASCompiler versiongcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0 MemcpyRunning memcpy benchmark memcpy: 3.18 GB/s (1 thread) ggml_mul_matRunning ggml_mul_mat benchmark with 4 threads ggml_init_cublas: found 1 CUDA devices: Model benchmarkRunning benchmark for all models
Running memcpy benchmark memcpy: 3.21 GB/s (heat-up) Running ggml_mul_mat benchmark with 4 threads ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no Running benchmark for all models
|
Jetson Orin Nano Developper Kit using cuBLASCompiler versiongcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 MemcpyRunning memcpy benchmark memcpy: 6.10 GB/s (1 thread) ggml_mul_matRunning ggml_mul_mat benchmark with 6 threads ggml_init_cublas: found 1 CUDA devices: Model benchmarkRunning benchmark for all models
|
Intel Core i5-8400 CPU + NVIDIA GeForce GTX 1070 with cuBLASCompiler versiongcc (conda-forge gcc 10.4.0-19) 10.4.0 nvcc: NVIDIA (R) Cuda compiler driver MemcpyRunning memcpy benchmark memcpy: 13.17 GB/s (1 thread) ggml_mul_matRunning ggml_mul_mat benchmark with 6 threads ggml_init_cublas: found 1 CUDA devices: Model benchmarkRunning benchmark for all models
|
|
Valve Jupiter (AMD Custom APU 0405) Update of the last time I ran this (last time was CPU only due to being on SteamOS): #89 (comment) CPU only
using iGPU
Didn't test anything more as I didn't have the disk space to download them. |
|
Hi there. These are some tests with a Mac Studio M1 Ultra with MacOS Sonoma 14.0.
|
|
@henry2man amazing results. What compiler options are you using / are you using ANE? |
I was simply experimenting with the project, so as far as I remember I just used the extras/all shell script without any other tweaks, just following the given instructions. If you want me to execute specific tests please tell me and I'll be glad to contribute the results here. EDIT: after a deeper research, I think I didn't make use of ANE, but I'm not 100% sure. |
|
Results for the new Raspberry Pi 5. Tests performed on a board with the active cooler.
These results are 4.5 to 6.2 times faster than the Raspberry Pi 4. NOTE: The packaged version of OpenBLAS has not been recompiled for the new CPU architecture, so it is about 50% slower than The |
|
CPU details: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Running benchmark for all models
|
|
Whats happening with commit 8a2bee6? Opi5 4gb |
|
@nickovs These are some very interesting results. Looking forward to the OpenBLAS results as well. @StuartIanNaylor The PP timing is the "prompt processing" time for a prompt of 256 tokens. As we transcribe with whisper, the context (i.e. the previously transcribed text) grows up to |
|
By way of comparison to the benchmarks I posted above, here is are the matrix multiplication numbers for the same Raspberry Pi 5 using OpenBLAS. It is notable that Whisper.cpp's native NEON code outperforms OpenBLAS on the Pi5 for everything except FP32, where OpenBLAS wins by some margin. I have not tried all the tuning options in OpenBLAS, but the options I did try didn't really change the performance compared to the pre-packaged version. |
I think this is where we benefit from ArmV8.2 and being a subgroup of Apple Silicon first-class citizen - optimized via ARM NEON. These results are 4.5 to 6.2 times faster than the Raspberry Pi 4. Linux ubuntu 6.6.0 #1 SMP PREEMPT Opi5 4GB performance governor Linux raspberrypi 6.1.0-rpi4-rpi-2712 Rpi5 4GB performance governor I dunno to be honest why Gflops is higher but whilst the |
|
@StuartIanNaylor Here is a straight up comparison of the same 54c978c commit between the Pi4 and the Pi5, both running the code compiled on the Pi4 on the Pi5 and then also recompiling the same commit on the Pi5.
This suggests that there is a little better than a 2-fold performance improvement on encode, and more like a 2.8 fold improvement on decode, just moving the code from the Pi4 to the Pi5. Recompiling on the Pi5 raises the encode performance to between 4.74 and 6.54 times faster that on the Pi4, but the decode performance remains only about 2.8 times faster than the Pi4 and doesn't benefit a great deal from the recompilation. (Note that this table hits GitHub's 10 column limit, so the decode speedup may not be displayed, but the numbers are in the comment source.) The key thing here as far as I'm concerned is that on the Pi5 the |
|
It would be great to have a test results db for this. I'm thinking similar to what DRM info does |
|
@jwinarske that would be great as maybe a seperate repo of fixed commits as we are not benching the software but the hardware. Linux ubuntu 6.6.0 #1 SMP PREEMPT Opi5 4GB performance governor 54c978c @nickovs Dunno as before as A76 gets vector mat/mul and the code is optimised for ArmV8,2+ that the poor Pi4 with openBlas was approx < 5 times slower than a RK3588s. |
"lib" is needed for windows. With this change, you can build whisper.cpp with OpenBLAS's prebuilt DLL. 1. extract a zip from https://github.com/xianyi/OpenBLAS/releases 2. copy the headers in (openblas)/include to the root directory of whisper.cpp 3. invoke cmake with -DCMAKE_LIBRARY_PATH=(openblas)\lib -DWHISPER_SUPPORT_OPENBLAS=ON 4. copy (openblas)/bin/libopenblas.dll to the same directory of whisper.dll after msbuild ggerganov/whisper.cpp#89 (comment)
Update whisper.cpp
|
Here is the result for NVIDIA GeForce GT 755M on Debian GNU/Linux 12 Bookworm using GCC 12.2.0 build with -DWHISPER_CLBLAST=ON: |
|
benchmark result with 11th Gen Intel Core(TM) i7-11700F @ 2.50GHz + Ubuntu 20.04 + gcc version 9.4.0
there is an impressive benchmark result(compare to above bench result in PC which was purchased by RMB12000(about USD 1700) a few years ago) with Xiaomi 14's powerful mobile SoC------Qualcomm SM8650-AB Snapdragon 8 Gen 3 (4 nm) + Xiaomi's HyperOS(derived from Android 14) + Android NDK r21e:
updated on 03-20-2024, Xiaomi 14 + Android NDK r26c( NDK r26c is required for special build optimization:https://github.com/cdeos/kantv/blob/master/external/whispercpp/CMakeLists.txt#L60)
|
|
Different results for different code commits - older version is much faster!CPU: AMD Ryzen 9 7950X3D 16-Core
system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0 whisper_print_timings: load time = 64.61 ms
system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | COREML = 0 | OPENVINO = 0 | whisper_print_timings: load time = 83.24 ms |
and others
Encoder
Collection of bench results for various platforms and devices.
If you want to submit info about your device, simply run the bench tool or the extra/bench-all.sh and report the results in the comments below.
Suggestions for better summary of the results are welcome
memcpy
MacBook M1 Pro
Ryzen 9 5950X
ggml_mul_mat
MacBook M1 Pro
Ryzen 9 5950X
The text was updated successfully, but these errors were encountered: