provided PTX was compiled with an unsupported toolchain

[thangld201]

./build/bin/main -m /tmp/ReluLLaMA-7B-PowerInfer-GGUF/llama-7b-relu.powerinfer.gguf -n 128 -t 8 --vram-budget 40 -p "Hi. How are you ?"
Log start
main: build = 1560 (2217e7f)
main: built with cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 for x86_64-linux-gnu
main: seed  = 1705391051
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: A100-SXM4-40GB, compute capability 8.0

CUDA error 222 at /py_lib/PowerInfer/ggml-cuda.cu:6466: the provided PTX was compiled with an unsupported toolchain.
current device: 0

I'm having the above error. Could you take a look ? @hodlen I used gpu with llama.cpp normally so this error should be from PowerInfer.

Similar issue with llama-cpp-python was raised here.

[hodlen]

Hi @thangld201, can I confirm that you compiled the code using the latest main branch, and encountered the error at this line?

The error might indicate that the PTX was generated by a compiler newer than what is supported by the CUDA driver and PTX JIT compiler. This line of code is involved in initializing the CUDA backend and is the same as in llama.cpp. Therefore, I suspect the issue may stem from your running environment.

I suggest trying again after updating the CUDA toolkit and ensuring that the compiler and driver versions match, as a similar issue was resolved in this way. Please feel free to provide more details if the error persists.