Can't run on CPU

[shanzhou2186]

I try to run the model on CPU in offline mode. But it depends on a package flash_attn, which needs to be compiled with nvcc on GPU.
I am wondering can we run this model on CPU.

[czczup]

This model has a total of 26B, is it difficult to run with the CPU?
For flash_attn, you can turn it off by modifying config.json in the model

Set L20 to "attn_implementation": "eager",

https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5/blob/main/config.json#L20

Set L200 to "use_flash_attn": false
https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5/blob/main/config.json#L200

[shanzhou2186]

I will try it. Thanks. If the dependency problem fixed, it should run on CPU successfully although it may be slow.