[Usage] Extremely Slow Inference Speed

[HuangZhen02]

Describe the issue

I run the LLAVA-34B model in one single A800(80GB) GPU. The inputs are some multi-modal math word problems(like those in MATH), but it turns out that each sample takes more than 1 minute.

I set do_sample=False and max_new_tokens=2048.