Performence on terminal is very different within llamachat

[Darkhell101]

I find the performence when using llama.cpp in terminal is better in llamachat.
The answer speed in terminal (interactive mode) is clearly faster on the same cpu usage level.
It's a little strange. Maybe it can be solved.

[alexrozanski]

thanks @fsrmlgd -- one thing might be to try tweaking the number of CPU cores used for prediction in LlamaChat (in Settings > General) if this is different to llama.cpp

[Darkhell101]

Thanks for your reply. The app is very helpful.
here is my test for two conditions.

llamachat.mp4

warp.mov

As you can see the cat on top bar, It seems ridiculous. I change nothing about default settings.

[Darkhell101]

If you need, some models can be directly used on PC are listed here.
https://huggingface.co/Pi3141
https://huggingface.co/eachadea
So make one easy to find a suitable version.

[alexrozanski]

thanks @fsrmlgd, will take a look into this!

[spacecowgoesmoo]

I set up vicuna7b on an m1 Mac mini and am seeing the same thing. Text in llamachat generates very slowly just like in the video above. Mac was at 75% cpu usage.

(and this is fine if llamas are still too heavy for low end systems, but that doesn’t seem like the case here.)

[alexrozanski]

@spacecowgoesmoo thanks, still getting round to this!

[Freccia]

Hello, here to say I experience the same issue. On terminal it works smoothly, but the GUI makes it very slow.
I am currently running on a Macbook Pro M1 with 16Gb of RAM.

btw amazing app design !