GPU not utilized with standard llama-cpp-python

[drphero]

When using loading the llava models, you can see that BLAS = 0 in the information printed in the console. This is because llama-cpp-python requires a special install if you want GPU capabilities. I'm not sure if this affects linux users or not, but it does affect windows users.

To properly install llama-cpp-python for nvidia GPUs:

pip install --no-cache-dir llama-cpp-python -C cmake.args="-DLLAMA_CUDA=ON" -vv

This of course requires Visual Studio to be installed with the "Desktop development with C++" option selected or the VS C++ Build Tools.

I found the --no-cache-dir necessary in order to get it to actually build it, so I'm not sure how this can be automatically done via the requirements.txt.

Now when using the llava models, you should see BLAS = 1 in the console.

[gokayfem]

we install it with pre-built wheels, it automatically supports cuda for linux and windows. recently pre-built wheel support added for macOS, i didnt add it to repo yet.