Is it possible to use quantization models with FULL GPU CUBLAS?

[Sing303]

Now when I try to use quantization models with FULL GPU CUBLAS, instead of recognizing text it writes nonsense, all kinds of signs instead of words.

[Sing303]

Examples

main -m ..\_models\large-v3.bin -f ..\_samples\kennedy.wav -bs 5
whisper_init_from_file_with_params_no_state: loading model from '..\_models\large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9
whisper_backend_init: using CUDA backend
whisper_model_load:     CUDA buffer size =  3094.88 MB
whisper_model_load: model size    = 3094.36 MB
whisper_backend_init: using CUDA backend
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   32.49 MB
whisper_init_state: compute buffer (encode) =  212.49 MB
whisper_init_state: compute buffer (cross)  =    9.45 MB
whisper_init_state: compute buffer (decode) =   99.30 MB

system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0 |

main: processing '..\_samples\kennedy.wav' (335854 samples, 21.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:12.000]   I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the earth.
[00:00:12.000 --> 00:00:21.000]   No single space project in this period will be more impressive to mankind or more important for the long-range exploration of space.


whisper_print_timings:     load time =  1392.50 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    11.99 ms
whisper_print_timings:   sample time =   120.13 ms /   315 runs (    0.38 ms per run)
whisper_print_timings:   encode time =   120.03 ms /     1 runs (  120.03 ms per run)
whisper_print_timings:   decode time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   batchd time =   773.98 ms /   313 runs (    2.47 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  2431.86 ms

main -m ..\_models\large-v3.bin -f ..\_samples\kennedy.wav -bs 5 -ng
whisper_init_from_file_with_params_no_state: loading model from '..\_models\large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9
whisper_model_load:      CPU buffer size =  3094.88 MB
whisper_model_load: model size    = 3094.36 MB
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   32.49 MB
whisper_init_state: compute buffer (encode) =  212.49 MB
whisper_init_state: compute buffer (cross)  =    9.45 MB
whisper_init_state: compute buffer (decode) =   99.30 MB

system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0 |

main: processing '..\_samples\kennedy.wav' (335854 samples, 21.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:12.000]   I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the earth.
[00:00:12.000 --> 00:00:21.000]   No single space project in this period will be more impressive to mankind or more important for the long-range exploration of space.


whisper_print_timings:     load time =  1526.86 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    15.42 ms
whisper_print_timings:   sample time =   136.39 ms /   321 runs (    0.42 ms per run)
whisper_print_timings:   encode time =  2324.60 ms /     1 runs ( 2324.60 ms per run)
whisper_print_timings:   decode time =    79.62 ms /     2 runs (   39.81 ms per run)
whisper_print_timings:   batchd time =  3729.08 ms /   317 runs (   11.76 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  7823.45 ms

main -m ..\_models\large-v3_80.bin -f ..\_samples\kennedy.wav -bs 5
whisper_init_from_file_with_params_no_state: loading model from '..\_models\large-v3_80.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 7
whisper_model_load: qntvr         = 2
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9
whisper_backend_init: using CUDA backend
whisper_model_load:     CUDA buffer size =  1656.39 MB
whisper_model_load: model size    = 1655.86 MB
whisper_backend_init: using CUDA backend
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   32.49 MB
whisper_init_state: compute buffer (encode) =  212.49 MB
whisper_init_state: compute buffer (cross)  =    9.45 MB
whisper_init_state: compute buffer (decode) =   99.30 MB

system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0 |

main: processing '..\_samples\kennedy.wav' (335854 samples, 21.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:30.000]  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


whisper_print_timings:     load time =   783.12 ms
whisper_print_timings:     fallbacks =   5 p /   0 h
whisper_print_timings:      mel time =    12.27 ms
whisper_print_timings:   sample time =  1867.29 ms /  5949 runs (    0.31 ms per run)
whisper_print_timings:   encode time =   121.02 ms /     1 runs (  121.02 ms per run)
whisper_print_timings:   decode time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   batchd time = 13015.51 ms /  5937 runs (    2.19 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time = 15809.39 ms

main -m ..\_models\large-v3_80.bin -f ..\_samples\kennedy.wav -bs 5 -ng
whisper_init_from_file_with_params_no_state: loading model from '..\_models\large-v3_80.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 7
whisper_model_load: qntvr         = 2
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9
whisper_model_load:      CPU buffer size =  1656.39 MB
whisper_model_load: model size    = 1655.86 MB
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   32.49 MB
whisper_init_state: compute buffer (encode) =  212.49 MB
whisper_init_state: compute buffer (cross)  =    9.45 MB
whisper_init_state: compute buffer (decode) =   99.30 MB

system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0 |

main: processing '..\_samples\kennedy.wav' (335854 samples, 21.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:12.000]   I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the earth.
[00:00:12.000 --> 00:00:21.000]   No single space project in this period will be more impressive to mankind or more important for the long-range exploration of space.


whisper_print_timings:     load time =   752.78 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    12.31 ms
whisper_print_timings:   sample time =   145.78 ms /   318 runs (    0.46 ms per run)
whisper_print_timings:   encode time =  2278.37 ms /     1 runs ( 2278.37 ms per run)
whisper_print_timings:   decode time =   102.37 ms /     4 runs (   25.59 ms per run)
whisper_print_timings:   batchd time =  3157.58 ms /   312 runs (   10.12 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  6453.02 ms

[bobqianic]

I am checking the cause of the problem.

[bobqianic]

Fixed