Llava-v1.5-7b ValueError: Unrecognized configuration class <class 'transformers.models.llava.configuration_llava.LlavaConfig'> for this kind of AutoModel: AutoModelForCausalLM.

[russell-tankl]

I am having trouble running Llava model in the benchmark suite. I am getting an error saying the LlavaConfig is unrecognized, but I see LlavaConfig in the choices.

Here is what I did:

• Set LLAVA_REPO_DIR to Llava repo, install transformers==4.37.2 and 4.38.0
• Ran all-in-one benchmark
• Also tried following requirements from Llava example (transformers==4.36.2, checkout llava repo with tags/v1.2.0) (
  https://github.com/intel-analytics/ipex-llm/tree/4b024b7aac4a2890af4a6753e130973ae9769e55/python/llm/example/GPU/PyTorch-Models/Model/llava
  )

Package                     Version
--------------------------- ------------------
accelerate                  0.21.0
annotated-types             0.6.0
antlr4-python3-runtime      4.9.3
bigdl-core-xe-21            2.5.0b20240409
bigdl-core-xe-esimd-21      2.5.0b20240409
certifi                     2024.2.2
charset-normalizer          3.3.2
colorama                    0.4.6
dpcpp-cpp-rt                2024.0.2
einops                      0.7.0
filelock                    3.13.4
fsspec                      2024.3.1
huggingface-hub             0.22.2
idna                        3.6
intel-cmplr-lib-rt          2024.0.2
intel-cmplr-lic-rt          2024.0.2
intel-extension-for-pytorch 2.1.10+xpu
intel-opencl-rt             2024.0.2
intel-openmp                2024.0.2
ipex-llm                    2.1.0b20240409
Jinja2                      3.1.3
MarkupSafe                  2.1.5
mkl                         2024.0.0
mkl-dpcpp                   2024.0.0
mpmath                      1.3.0
networkx                    3.3
numpy                       1.26.4
omegaconf                   2.3.0
onednn                      2024.0.0
onemkl-sycl-blas            2024.0.0
onemkl-sycl-datafitting     2024.0.0
onemkl-sycl-dft             2024.0.0
onemkl-sycl-lapack          2024.0.0
onemkl-sycl-rng             2024.0.0
onemkl-sycl-sparse          2024.0.0
onemkl-sycl-stats           2024.0.0
onemkl-sycl-vm              2024.0.0
packaging                   24.0
pandas                      2.2.1
pillow                      10.3.0
pip                         23.3.1
protobuf                    5.26.1
psutil                      5.9.8
py-cpuinfo                  9.0.0
pydantic                    2.7.0b1
pydantic_core               2.18.0
python-dateutil             2.9.0.post0
pytz                        2024.1
PyYAML                      6.0.1
regex                       2023.12.25
requests                    2.31.0
safetensors                 0.4.2
sentencepiece               0.2.0
setuptools                  68.2.2
six                         1.16.0
sympy                       1.12.1rc1
tabulate                    0.9.0
tbb                         2021.12.0
tokenizers                  0.15.2
torch                       2.1.0a0+cxx11.abi
torchvision                 0.16.0a0+cxx11.abi
tqdm                        4.66.2
transformers                4.36.2
typing_extensions           4.11.0
tzdata                      2024.1
urllib3                     2.2.1
wheel                       0.41.2

Traceback (most recent call last):
  File "E:\<username>\<path_to_repo>\python\llm\dev\benchmark\all-in-one\run.py", line 1845, in <module>
    run_model(model, api, in_out_pairs, conf['local_model_hub'], conf['warm_up'], conf['num_trials'], conf['num_beams'],
  File "E:\<username>\<path_to_repo>\python\llm\dev\benchmark\all-in-one\run.py", line 99, in run_model
    result = run_transformer_int4_fp16_gpu_win(repo_id, local_model_hub, in_out_pairs, warm_up, num_trials, num_beams, low_bit, cpu_embedding, batch_size, streaming)
  File "E:\<username>\<path_to_repo>\python\llm\dev\benchmark\all-in-one\run.py", line 961, in run_transformer_int4_fp16_gpu_win
    model = AutoModelForCausalLM.from_pretrained(model_path, load_in_low_bit=low_bit, optimize_model=True,
  File "C:\Users\<username>\Miniconda3\envs\ipex_llm_bench\lib\site-packages\ipex_llm\transformers\model.py", line 330, in from_pretrained
    model = cls.load_convert(q_k, optimize_model, *args, **kwargs)
  File "C:\Users\<username>\Miniconda3\envs\ipex_llm_bench\lib\site-packages\ipex_llm\transformers\model.py", line 444, in load_convert
    model = cls.HF_Model.from_pretrained(*args, **kwargs)
  File "C:\Users\<username>\Miniconda3\envs\ipex_llm_bench\lib\site-packages\transformers\models\auto\auto_factory.py", line 569, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.llava.configuration_llava.LlavaConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, ElectraConfig, ErnieConfig, FalconConfig, FuyuConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, LlamaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MptConfig, MusicgenConfig, MvpConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, LlavaConfig, LlavaMptConfig, LlavaMistralConfig.

[pengyb2001]

If you need to run the all-in-one benchmark, please first configure the environment using https://github.com/liu-shaojun/ipex-llm/tree/3f20072736ebe2caae6793a8c1a7c092b9627ba5/python/llm/example/GPU/PyTorch-Models/Model/llava (which uses LLaVA version 1.1.1); support for LLaVA v1.2 and transformers 4.36 in the benchmark script will be updated later;
Additionally, running the example directly with v1.2 and transformers 4.36 should pose no issues.