feedback - 7b models crashing

[Ar57m]

hey, I am running your app on a Galaxy M52, 3B models runs fine on it, but 7b as pygmalion2 7b(3ks) and mystral pygmalion 7b(3km) it crashes and closes the app, on llama.cpp through termux it runs around 0.1~0.2 tokens/second.

[sfiannaca]

In my test, this bug occurs when the model surpasses the hardware capacity (out of memory).

It seems that Termux utilizes less RAM compared to Flutter apps.

You can attempt to utilize Q2_K, but the token speed of the 7B model is still relatively slow on medium-spec smartphone

[Ar57m]

thanks for the response and for the app.

Suggestion: if you can add more parameters like temperature, repetition penalty etc.

I've notice that sometimes after some conversations, some messages are fully generated and then disappear(become empty).
And sometimes I close the app and enter again on it, it doesn't want to generate the response or crashes.

[Ar57m]

forgot to mention I'm running the version 1.0.3 on android.

[sfiannaca]

The app is currently under development, and I'm just a user like you, not the author

[danemadsen]

Try the new build, Ive exposed both those parameters.

[danemadsen]

Yeah certain 7B models crash the app because they use too much memory. In my testing if you're going to attempt to use 7B models you should try to use smaller Quant's.

[Ar57m]

I've noticed when I switch models, the app cache accumulate the models unnecessary.

[danemadsen]

Ill look into it

[Ar57m]

thanks for the update, the new release(maid 1.0.5 apk) it's very stable now, I can run pygmalion2 7b 3ks slowly and without any crashes, thanks again

[danemadsen]

That's great. Let me know if there's anything else I can do to improve the app.

[Ar57m]

The app is awesome now(1.0.8) without the giant cache and no more crashes, good work guys. 👏

Some suggestions: Support for llava models on mobile, min-p(parameter), some presets(character tab) for chatml, llama2, vicuna, alpaca prompt formating and some options to customize the background picture, AI 's picture, user's picture(like the aesthetic ui from koboldcpp).

[Ar57m]

Vali is developing a front-end like ST for android https://github.com/Vali-98/ChatterUI

[danemadsen]

Good luck to him. He's using react native though and I'm using flutter. Different workflows.

[Ar57m]

can you please update the llama.cpp of the app to support stablelm models?

[Ar57m]

I've notice you have updated llama, but tried to run a stablelm3b model on the 1.1.2 version and is crashing

[danemadsen]

You need to use the latest actions release https://github.com/MaidFoundation/Maid/actions/workflows/main.yml Until I do another fixed point release. With regards, Dane Madsen

…

________________________________ From: Ar57m ***@***.***> Sent: Monday, November 20, 2023 3:01:17 AM To: MaidFoundation/Maid ***@***.***> Cc: Dane Madsen ***@***.***>; State change ***@***.***> Subject: Re: [MaidFoundation/Maid] feedback - 7b models crashing (Discussion #28) I've notice you have updated llama, but tried to run a stablelm3b model on the 1.1.2 version and is crashing — Reply to this email directly, view it on GitHub<#28 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACYA2IZO76R7WWKIJZRJIXTYFI3N3AVCNFSM6AAAAAA6RTHZTOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMMJSGE3DO>. You are receiving this because you modified the open/close state.Message ID: ***@***.***>