想知道开发维护人员一般用HuggingFaceCausalLM还是VLLM

[IcyFeather233]

VLLM有时候会卡住，如：#1023 (comment) ，也会有一些别的怪问题，总之不是很稳定

但是HuggingFaceCausalLM比VLLM慢好多感觉

[bittersweet1999]

用LMdeploy

[IcyFeather233]

from opencompass.models import LmdeployPytorchModel 这样吗

使用的时候有什么讲究呢，和用HuggingFaceCausalLM相比，直接把type换成这个就行吗

[bittersweet1999]

https://github.com/open-compass/opencompass/blob/main/configs/eval_internlm_chat_lmdeploy_pytorch.py
https://github.com/open-compass/opencompass/blob/main/configs/eval_internlm_chat_turbomind.py
供参考

[IcyFeather233]

是否LMDeploy还没有支持MixtralForCausalLM类型的模型？

# mixtral moe
_meta_template = dict(
    begin="<s>",
    round=[
        dict(role="HUMAN", begin='[INST]', end='[/INST]'),
        dict(role="BOT", begin="", end='</s>', generate=True),
    ],
    eos_token_id=2
)

GPU_NUMS = 4


models = [dict(
    type=TurboMindModel,
    abbr='mixtral_moe',
    path='mixtral_moe',
    engine_config=dict(session_len=2048,
                       max_batch_size=16),
    gen_config=dict(top_k=1,
                    top_p=0.8,
                    temperature=1.0,
                    max_new_tokens=100),
    max_out_len=100,
    max_seq_len=2048,
    batch_size=8,
    concurrency=8,
    meta_template=_meta_template,
    run_cfg=dict(num_gpus=GPU_NUMS, num_procs=1),
    end_str='</s>',
)
]

报错说

File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/utils.py", line 54, in get_model_from_config
    return ARCH_MAP[arch]
KeyError: 'MixtralForCausalLM'

详细报错

04/12 10:37:13 - OpenCompass - INFO - Task [lc_0219_online_nosafe_2_official_hf/lukaemon_mmlu_professional_law_0]
2024-04-12 10:37:43,491 - lmdeploy - WARNING - model_source: hf_model
Ignored unknown kwarg option __type
Ignored unknown kwarg option __type
Ignored unknown kwarg option __type
Ignored unknown kwarg option __type
Ignored unknown kwarg option __type
Ignored unknown kwarg option __type
Traceback (most recent call last):
  File "/maindata/data/shared/Security-SFT/cmz/opencompass/opencompass/tasks/openicl_infer.py", line 153, in <module>
    inferencer.run()
  File "/maindata/data/shared/Security-SFT/cmz/opencompass/opencompass/tasks/openicl_infer.py", line 65, in run
    self.model = build_model_from_cfg(model_cfg)
  File "/maindata/data/shared/Security-SFT/cmz/opencompass/opencompass/utils/build.py", line 25, in build_model_from_cfg
    return MODELS.build(model_cfg)
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/maindata/data/shared/Security-SFT/cmz/opencompass/opencompass/models/turbomind.py", line 64, in __init__
    tm_model = TurboMind.from_pretrained(path, engine_config=engine_config)
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 398, in from_pretrained
    return cls(model_path=pretrained_model_name_or_path,
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 167, in __init__
    self.model_comm = self._from_hf(model_source=model_source,
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 259, in _from_hf
    model_arch = get_model_from_config(model_path)
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/utils.py", line 54, in get_model_from_config
    return ARCH_MAP[arch]
KeyError: 'MixtralForCausalLM'
[2024-04-12 10:37:45,718] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 40227) of binary: /maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/bin/python
Traceback (most recent call last):
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/maindata/data/shared/Security-SFT/cmz/opencompass/opencompass/tasks/openicl_infer.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-04-12_10:37:45
  host      : dsw-3043-d7655499-nzk42
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 40227)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

[bittersweet1999]

看起来是的，lmdeploy好像mistral和mixtral都还未支持，具体可以查看lmdeploy那边的issues