Android example app

[ggerganov]

Implement a very basic Java application using whisper.cpp. It can be used as an example for running Whisper on Android.

The ggwave-java project can be used as a good starting point. It already provides the audio capture functionality. Instead of passing it to ggwave, we just need to pass it to whisper.cpp.

Edit:
Looking for volunteers to help with this - ideally, we would like to have the same functionality demonstrated as in the iOS example application.

---

Update:

Thanks to @Digipom - this is now available!
Checkout the examples/whisper.android example

[Hannes1]

https://github.com/alphacep/vosk-android-demo

Maybe could also have a look at vosk demo, it's more adapted for streaming though but just for example...

This is a bit off topic, I've been playing around with react native and wenet also, haven't done IOS yet but yeah the android part is working:
https://github.com/Hannes1/react-native-wenet

[luke-jr]

Maybe one of the open source Android keyboard projects would be a good candidate for integration :)

[CristhianNY]

it's possible to hear the Spanish language? , I have been trying but i speak in Spanish but I'm no able to transcribe spanish

[ArthurPeabody]

me, too.

[rpdrewes]

I use whisper.cpp from my de-googled Android phone daily for months now for dictation to SMS and email and web search. My preferred method is to use the Konele app as the voice input frontend which sends the audio to my own private server which is running a very simple python wrapper around whisper.cpp. Here is my project: https://github.com/rpdrewes/whisper-websocket-server

Another alternative is this port of whisper.cpp to run natively on Android. I have tested it and it also works well but it is not as fast as my server-based approach in my environment (but if you have a slower server and/or a faster phone it might be better for you). It uses Konele as a frontend for the input as well, I believe. See: https://github.com/alex-vt/WhisperInput

[zhouwg]

Thanks for the great whisper.cpp although I heard it too late.

I had been finished a similar PoC with Mozilla's DeepSpeech on Android-based phone.

But I'd like to try to integrate whisper.cpp with KanTV because it's more closer to "real application".

The following snapshot will be a start point and there's a significant performance issue(could Google's Highway be used to solve performance issue?)

[zhouwg]

integrate whisper.cpp to KanTV step1:

How to practise/play with this branch:

adb logcat | grep KANTV
(logs in Java layer / JNI layer / Native layer would be displayed with same prefix properly, so it's helpful for troubleshooting and tracking source codes and performance bottleneck analysis in whisper.cpp)

[liam-mceneaney]

I added a streaming example in a fork: https://github.com/liam-mceneaney/androidwhisper.cpp/tree/master

Also my PR: #1924

[zhouwg]

thanks so much. It's very useful for other Android developer and it would be very helpful for my on-going PoC(which just for personal interests/hobbies).

but I couldn't build it and then study your code accordingly because of GFW(I guess it's because of it's highly sensitive period in China now)

btw, could ggerganov create/open a WeChat account(tip:many personal privacy information(include face identification) might be required/provided to open a WeChat account and you might be known we are used to that) so I can buy some coffee which express my sincerely thanks for your great whisper.cpp. I guess you should receive a lot ZANSHANG(aka "赞赏", or "reward" ------- similar to Paypal's personal donation) from programmers in China if you have a WeChat account.

[zhouwg]

@liam-mceneaney, thanks so much. I got your key-point of code:

//streaming attempt:
JNIEXPORT void JNICALL
Java_com_whispercpp_whisper_WhisperLib_00024Companion_fullStreamTranscribe(
        JNIEnv *env, jobject thiz, jlong context_ptr, jint num_threads, jfloatArray audio_data) {
    UNUSED(thiz);
    struct whisper_context *context = (struct whisper_context *) context_ptr;
    jfloat *audio_data_arr = (*env)->GetFloatArrayElements(env, audio_data, NULL);
    const jsize audio_data_length = (*env)->GetArrayLength(env, audio_data);

    // The below adapted from the Objective-C iOS sample
    struct whisper_full_params params = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
    params.print_realtime = true;
    params.print_progress = false;
    params.print_timestamps = true;
    params.print_special = false;
    params.translate = false;
    params.language = "en";
    params.n_threads = num_threads; //how many threads can I use on an S23?
    //potentially use an initial prompt for custom vocabularies?
    // initial_prompt: Optional[str]
    //        Optional text to provide as a prompt for the first window. This can be used to provide, or
    //        "prompt-engineer" a context for transcription, e.g. custom vocabularies or proper nouns
    //        to make it more likely to predict those word correctly.
    //params.initial_prompt = "Transcription of Tactical Combat Casualty Drugs such as Fentanyl, Ibuprofen, Amoxicillin, Epinephrine, TXA, Hextend, Ketamine, Oral Transmucosal Fentanyl Citrate. ";
    params.offset_ms = 0;
    params.no_context = true;
    params.single_segment   = true; //hard code for true, objc example has it based on a button press
    params.no_timestamps    = params.single_segment; //from streaming objc example

    whisper_reset_timings(context);

    LOGI("About to run whisper_full");
    if (whisper_full(context, params, audio_data_arr, audio_data_length) != 0) {
        LOGI("Failed to run the model");
    } else {
        whisper_print_timings(context);
    }
    (*env)->ReleaseFloatArrayElements(env, audio_data, audio_data_arr, JNI_ABORT);
}

Thanks so much.

[zhouwg]

ASR/transcription performance on Xiaomi 14 is about 5x-20x better then other Android phone(low-end phone from vivo, huawei), but it's still not enough for real-time subtitle with online TV.

transcription performance can be improved by about 1-3 seconds when enable openblas.

performance of mulmat benchmark seems be improved a lot / significantly when enable openblas.

so I guess Apple's dedicated machine learning acceleration library might be very important for performance on iOS/Mac.just like Georgi Gerganov said before. and we should/might study something about Qualcomm's dedicated machine learning acceleration library accordingly.

updated on 03-10-2024(2024-03-10, 15:46, Beijing Time / GMT + 8),

updated on 03-10-2024(2024-03-10, 23:41 Beijing Time / GMT + 8):

I think I got the point although it's still not enough for real-time subtitle with online TV.

updated on 03-11-2024(2024-03-11,10:40) Beijing Time / GMT + 8

less than 2 seconds for the first time.

commit could be found here. FYI.

updated on 03-12-2024(2024-03-12,00:51 , Beijing Time / GMT +8)
(12:45 means 00:45 in 03-12-2024)

less than 1 second for the first time.

AI is really a magic technology. thanks for great whisper.cpp.

---

there is a very concise implementation of Android example during my on-going PoC.

some source codes could be found here:

https://github.com/cdeos/kantv/blob/kantv-poc-with-whispercpp/external/whispercpp/whisper.cpp#L6726

https://github.com/cdeos/kantv/blob/kantv-poc-with-whispercpp/external/whispercpp/whisper.h#L620

https://github.com/cdeos/kantv/blob/kantv-poc-with-whispercpp/external/whispercpp/jni/whispercpp-jni.c

https://github.com/cdeos/kantv/blob/kantv-poc-with-whispercpp/cdeosplayer/cdeosplayer-lib/src/main/java/org/ggml/whispercpp/whispercpp.java

https://github.com/cdeos/kantv/blob/kantv-poc-with-whispercpp/cdeosplayer/kantv/src/main/java/com/cdeos/kantv/ui/fragment/ASRResearchFragment.java

if it's considered well and accepted, I'd like to spent time(clean up codes and remove any dependence......, just follow whisper.cpp's principle: without dependencies, keep clean, ...) to merge code from there to here and submit a PR accordingly.
if not, I'll skip it.
thanks so much.

[zhouwg]

updated on 03-16-2024,22:36 (Beijing Time / GMT + 8),

real-time English subtitle for online TV by whisper.cpp

kantv-realtime-subtitle-by-whispercpp-demo-on-xiaomi14.mp4

[zhouwg]

updated on 03-24-2024, 22:55,

@ggerganov, better performance with better stability after finetune with method which introduced in #1951, but sometimes whisper.cpp will produce meaningless repeat tokens.

BTW, the root cause of the crash was that there were some invalid UTF-8 strings coming from whisper.cpp.

https://stackoverflow.com/questions/60722231/jni-detected-error-in-application-input-is-not-valid-modified-utf-8-illegal-st

zhouwg/kantv#97

realtime-subtitle-by-whispercpp-demo-on-xiaomi14-finetune-20240324.mp4