Issue with Transcription on Apple's ARM Systems Compatible Core-ML Model (WHISPER_COREML=1 make -j)

[EthanKeng]

Hello,
I am using Macbook Air M2.

I've encountered an issue where I can successfully run a normal model without any problems (make tiny). However, when attempting to run Apple's ARM systems compatible Core-ML (model-ggml-tiny.en-encoder.mlmodelc, and build the core-ml mode by WHISPER_COREML=1 make -j
), the transcription results are incorrect. This is evident in the sample audio (originally 'jfk.wav,' renamed to 'sample.wav'). The output is significantly inaccurate.

I'm curious if anyone else has faced a similar problem.

Here is the output:

(py310-whisper) yc.keng@ycks-air whisper.cpp % arch -arm64 ./main -m models/ggml-tiny.en.bin -l auto -otxt -f samples/sample.wav -of sample/sample_output
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-tiny.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 384
whisper_model_load: n_audio_head  = 6
whisper_model_load: n_audio_layer = 4
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 384
whisper_model_load: n_text_head   = 6
whisper_model_load: n_text_layer  = 4
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 1 (tiny)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2
ggml_metal_init: picking default device: Apple M2
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/yc.keng/code/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M2
ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 17179.89 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    73.61 MiB, (   74.36 / 16384.02)
whisper_model_load:    Metal buffer size =    77.18 MB
whisper_model_load: model size    =   77.11 MB
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2
ggml_metal_init: picking default device: Apple M2
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/yc.keng/code/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M2
ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 17179.89 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     7.88 MiB, (   82.23 / 16384.02)
whisper_init_state: kv self size  =    8.26 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     8.80 MiB, (   91.03 / 16384.02)
whisper_init_state: kv cross size =    9.22 MB
whisper_init_state: loading Core ML model from 'models/ggml-tiny.en-encoder.mlmodelc'
whisper_init_state: first run on a device may take a while ...
whisper_init_state: Core ML model loaded
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, (   91.06 / 16384.02)
whisper_init_state: compute buffer (conv)   =    4.97 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, (   91.08 / 16384.02)
whisper_init_state: compute buffer (cross)  =    4.01 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, (   91.09 / 16384.02)
whisper_init_state: compute buffer (decode) =   96.02 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     3.12 MiB, (   94.20 / 16384.02)
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     2.20 MiB, (   96.39 / 16384.02)
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    89.95 MiB, (  186.33 / 16384.02)
main: WARNING: model is not multilingual, ignoring language and translation options

system_info: n_threads = 4 / 8 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 1 | OPENVINO = 0 | 

main: processing 'samples/sample.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:30.000]   And so, my fellow, my fellow and thank you, thank you, very, thank you, thank, thank, he, thank, thank, thank, Chairman, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank, thank

output_txt: failed to open 'sample/sample_output.txt' for writing

whisper_print_timings:     load time =   146.73 ms
whisper_print_timings:     fallbacks =   5 p /   1 h
whisper_print_timings:      mel time =     8.81 ms
whisper_print_timings:   sample time =  2182.64 ms /  6438 runs (    0.34 ms per run)
whisper_print_timings:   encode time =    59.43 ms /     1 runs (   59.43 ms per run)
whisper_print_timings:   decode time =    25.44 ms /     6 runs (    4.24 ms per run)
whisper_print_timings:   batchd time =  8384.28 ms /  6408 runs (    1.31 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time = 10957.63 ms
ggml_metal_free: deallocating
ggml_metal_free: deallocating

[ggerganov]

It works on my M2 Ultra:

make clean
WHISPER_COREML=1 make -j && ./main -m models/ggml-tiny.en.bin -f samples/jfk.wav

 WHISPER_COREML=1 make -j && ./main -m models/ggml-tiny.en.bin -f samples/jfk.wav 
I whisper.cpp build info: 
I UNAME_S:  Darwin
I UNAME_P:  arm
I UNAME_M:  arm64
I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -DGGML_USE_ACCELERATE -DGGML_USE_METAL
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -pthread -DWHISPER_USE_COREML -DGGML_USE_METAL
I LDFLAGS:   -framework Accelerate -framework Foundation -framework CoreML -framework Foundation -framework Metal -framework MetalKit
I CC:       Apple clang version 15.0.0 (clang-1500.0.40.1)
I CXX:      Apple clang version 15.0.0 (clang-1500.0.40.1)

make: Nothing to be done for `default'.
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-tiny.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 384
whisper_model_load: n_audio_head  = 6
whisper_model_load: n_audio_layer = 4
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 384
whisper_model_load: n_text_head   = 6
whisper_model_load: n_text_layer  = 4
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 1 (tiny)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Ultra
ggml_metal_init: picking default device: Apple M2 Ultra
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/ggerganov/development/github/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M2 Ultra
ggml_metal_init: GPU family: MTLGPUFamilyApple8  (1008)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 154618.82 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    73.61 MiB, (   75.23 / 147456.00)
whisper_model_load:    Metal total size =    77.18 MB (1 buffers)
whisper_model_load: model size    =   77.11 MB
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Ultra
ggml_metal_init: picking default device: Apple M2 Ultra
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/ggerganov/development/github/whisper.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M2 Ultra
ggml_metal_init: GPU family: MTLGPUFamilyApple8  (1008)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 154618.82 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     7.88 MiB, (   83.11 / 147456.00)
whisper_init_state: kv self size  =    8.26 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     8.80 MiB, (   91.91 / 147456.00)
whisper_init_state: kv cross size =    9.22 MB
whisper_init_state: loading Core ML model from 'models/ggml-tiny.en-encoder.mlmodelc'
whisper_init_state: first run on a device may take a while ...
whisper_init_state: Core ML model loaded
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, (   91.92 / 147456.00)
whisper_init_state: compute buffer (conv)   =    4.97 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, (   91.94 / 147456.00)
whisper_init_state: compute buffer (cross)  =    4.01 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, (   91.95 / 147456.00)
whisper_init_state: compute buffer (decode) =   96.02 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     3.12 MiB, (   95.06 / 147456.00)
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     2.20 MiB, (   97.25 / 147456.00)
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    89.95 MiB, (  187.19 / 147456.00)

system_info: n_threads = 4 / 24 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 1 | OPENVINO = 0 | 

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:07.960]   And so my fellow Americans ask not what your country can do for you
[00:00:07.960 --> 00:00:10.760]   ask what you can do for your country.


whisper_print_timings:     load time =   108.59 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     7.22 ms
whisper_print_timings:   sample time =    33.97 ms /   140 runs (    0.24 ms per run)
whisper_print_timings:   encode time =    82.34 ms /     1 runs (   82.34 ms per run)
whisper_print_timings:   decode time =     6.32 ms /     1 runs (    6.32 ms per run)
whisper_print_timings:   batchd time =    88.01 ms /   135 runs (    0.65 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =   419.41 ms
ggml_metal_free: deallocating
ggml_metal_free: deallocating

Have you tried to regenerate the Core ML model?

./models/generate-coreml-model.sh tiny.en

[EthanKeng]

Thank you @ggerganov.
I upgraded my OS to Sonoma this morning without doing anything else and it fixed 👍