Flash Attention

[ss4elby]

So I noticed it runs WAY slow, then realized my card was not set up for that, I am running ye oldie p40. So no tensor cores. But this fellow over at flash attention apparently made it possible to work without them? ggerganov#7188 I assume this in not implemented for this yet, any chance?

[LostRuins]

No, it's not implemented yet. I will merge it for the next version

[ss4elby]

Appreciated, your work is something amazing!

[Spacellary]

Truly a joyous occasion! This looks very promising!

[LostRuins]

Hi, can see if this works fine for you on the latest version?

[gustrd]

I checked with my old MX150 and now it works.

The llama.cpp upgrade to CUDA without tensor cores must have solved it. The prompt processing speed is higher now (around 2x faster), but the generation a bit slower (around 20%). But this is a good tradeoff in the end.

[ss4elby]

It seems to work fine, holy hell its quick too. Thank you!