GPU acceleration on android

[Ar57m]

If possible, I don't really understand anything about it, it's just an idea, to use opencl/clblast if it helps the speeds 🙂

[danemadsen]

Yeah im not sure if its possible and if it is i certainly think its out of my depth of knowledge. I would love it if someone could implement it though.

[sfiannaca]

I came across these old issues, but I'm unsure if they still work. It seems like they just need some adjustments in CMake.

ggerganov/llama.cpp#2169

beyond what is already reported here
https://github.com/ggerganov/llama.cpp#building-the-project-using-termux-f-droid

[Ar57m]

some time ago I broke my termux 5 times trying that, probably I did something wrong

[danemadsen]

If its as easy as that guide it shouldn't be hard to get working. I'll have a look at it later in the week maybe.

[Ar57m]

by the way, you can ask a space on koboldai discord to spread the word

[danemadsen]

I don't have reddit or discord but I could probably make a throwaway to spread the word.

[Ar57m]

Some suggestions: to implement a button to use the native tts(google/samsung/etc) to read the AI message, an option to edit the AI's message, min P parameter and an option to change the background image.

[sfiannaca]

It could be a good idea, but currently, I don't see any Text-to-Speech package good enough on Flutter

I found only this one, but using native TTS feels so unnatural

flutter_tts
Demo:
https://dlutton.github.io/flutter_tts

[danemadsen]

I havnt bothered implementing TTS because most android keyboards already have a button for it.

[Ar57m]

Ok np, I tried to select the text to speak but the option is not appearing like on duckduckgo appears

[sfiannaca]

Stock Android keyboards typically feature a Speak-to-Text function not Text-to-Speak

From your screenshots, it appears to be a browser-specific feature

For native Text-to-Speech (TTS), you can enable accessibility settings with Android devices but it's not practical

[Ar57m]

STT works fine on my keyboard, I mean TTS, I have an app installed which I select text anywhere it appears the option to speech, but in maid it doesn't appear, even the option to share the text, idk but maybe it doesn't work on flutter, the app seems to have its own buttons(select text,copy, paste)

[abasgames]

maybe vulkan acceleration?
https://github.com/nomic-ai/llama.cpp

[danemadsen]

im open to it

[abasgames]

it would be also nice to use the opencl aswell though clblast
mlc llm already uses that for acceleration on android

[abasgames]

anyone feel free to pick up my mess and try to get it working
https://github.com/abasgames/maidvlk/actions/

[danemadsen]

anyone feel free to pick up my mess and try to get it working
https://github.com/abasgames/maidvlk/actions/

Haha yeah I saw it. Good effort for trying.

[Foul-Tarnished]

You can probably use NNAPI
https://developer.android.com/ndk/guides/neuralnetworks

this can use it to get faster
https://github.com/ShiftHackZ/Stable-Diffusion-Android

[danemadsen]

OK as of 98533ea Ive enabled Vulkan Acceleration on Linux and android (windows is coming too). Ive tested it out on android,it seems noticeably faster but not jaw droppingly.

[Ar57m]

Nice good work, it's hard to tell the speeds, if was possible to have a buttom to desactive&active vulkan, and have more info about the speeds in the log (in the settings) tks/s processing and generation

[qnixsynapse]

You should provide an option to disable it. Some GPUs, both on the desktop and especially on Android, do not like it.

[danemadsen]

its currently disabled in the latest release and ive found that even when it is enabled it doesn't really speed things up that much

[qnixsynapse]

Yeah.. Also, it's better to use SYCL or CUDA or ROCM in place of Vulkan on desktops and fallback to CPU when an error occurs. SYCL can run on all three (currently llamacpp SYCL can run on intel and nvidia, not sure about AMD but it should by design, work on AMD as well).