[Bug] Qwen1.5-32B在做awq量化并convert的时候会报错

[dingjingzhen]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.

Describe the bug

我是在4卡A10上做的，当group_size=128的时候会报错，报错内容是 "intermediate_size": 27392,无法整除4*128，当我将group size改成64，awq和convert成功，跑benchmark的generate脚本时报错 unsupported group size

Reproduction

step1:

export HF_MODEL=/workspace/mnt/models/Qwen1.5-32B-Chat
export WORK_DIR=/workspace/mnt/models/Qwen1.5-32B-Chat/awq_lmdeploy_g64

lmdeploy lite auto_awq \
   $HF_MODEL \
  --calib-dataset 'ptb' \
  --calib-samples 2 \
  --calib-seqlen 2048 \
  --w-bits 4 \
  --w-group-size 64 \
  --work-dir $WORK_DIR

step2:

lmdeploy convert \
    qwen \
    /workspace/mnt/models/Qwen1.5-32B-Chat/awq_lmdeploy_g64 \
    --model-format awq \
    --group-size 64 \
    --tp 4

step3:
python3 profile_generation.py ./workspace/ --tp 4 --model-format awq --cache-max-entry-count 0.5

Environment

pytorch 2.1.2 cuda12.1

Error traceback

No response

[AllentDan]

是一定想要 tp 4 吗？ tp 2，然后 group size 不变 启两个服务不行吗

[dingjingzhen]

是一定想要 tp 4 吗？ tp 2，然后 group size 不变 启两个服务不行吗

嗯，我们比较重视首字延时，4卡按道理肯定比2卡快

[AllentDan]

turbomind 目前只支持 group size 128 推理。"intermediate_size": 27392,无法整除4*128 报错是发生在什么阶段

[github-actions]

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.