请教，冻结一些层finetune的int8模型如何推断。

[dayu1979]

好像用infer_finetuning.py加载微调模型，直接报错。出错信息如下

[dayu1979]

/mnt/d/aitools/nlg/chatglm2_finetuning/infer_finetuning.py:41 in │
│ │
│ 38 │ │ train_weight = './best_ckpt/last/best.pt' │
│ 39 │ │
│ 40 │ #加载微调权重 │
│ ❱ 41 │ pl_model.load_sft_weight(train_weight,strict=True) │
│ 42 │ #pl_model.load_sft_weight('./best_ckpt/best.pt',is_trainable=True) │
│ 43 │ model = pl_model.get_llm_model() │
│ 44 │ #保存hf权重 │
│ │
│ /home/dayu/miniconda3/lib/python3.10/site-packages/deep_training/trainer/pl/modelweighter.py:130 │
│ in load_sft_weight │
│ │
│ 127 │ │ │ self: nn.Module │
│ 128 │ │ │ h = self.register_load_state_dict_post_hook(assert_state_dict_fn) │
│ 129 │ │ │ # TransformerBase类 可能有自定义额外模块 │
│ ❱ 130 │ │ │ self.load_state_dict(weights_dict_new, strict=strict) │
│ 131 │ │ │ h.remove() │
│ 132 │
│ 133 │
│ │
│ /home/dayu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:2027 in │
│ load_state_dict │
│ │
│ 2024 │ │ │ │ │ "it should be done inplace." │
│ 2025 │ │ │ │ ) │
│ 2026 │ │ │
│ ❱ 2027 │ │ load(self, state_dict) │
│ 2028 │ │ del load │
│ 2029 │ │ │
│ 2030 │ │ if strict: │
│ │
│ /home/dayu/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:2020 in load │
│ │
│ 2017 │ │ │ # Note that the hook can modify missing_keys and unexpected_keys. │
│ 2018 │ │ │ incompatible_keys = _IncompatibleKeys(missing_keys, unexpected_keys) │
│ 2019 │ │ │ for hook in module.load_state_dict_post_hooks.values(): │
│ ❱ 2020 │ │ │ │ out = hook(module, incompatible_keys) │
│ 2021 │ │ │ │ assert out is None, ( │
│ 2022 │ │ │ │ │ "Hooks registered with register_load_state_dict_post_hook are no │
│ 2023 │ │ │ │ │ "expected to return new values, if incompatible_keys need to be modi │
│ │
│ /home/dayu/miniconda3/lib/python3.10/site-packages/deep_training/trainer/pl/modelweighter.py:120 │
│ in assert_state_dict_fn │
│ │
│ 117 │ │ │ │ │ │ print('Error in load_sft_weight missing_keys',missing_keys) │
│ 118 │ │ │ │ if incompatible_keys.unexpected_keys: │
│ 119 │ │ │ │ │ if strict: │
│ ❱ 120 │ │ │ │ │ │ raise ValueError('Error in load_sft_weight unexpected_keys', inc │
│ 121 │ │ │ │ │ else: │
│ 122 │ │ │ │ │ │ print(('Error in load_sft_weight unexpected_keys', incompatible │
│ 123 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: ('Error in load_sft_weight unexpected_keys', ['_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.0.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.0.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.0.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.0.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.1.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.1.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.1.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.1.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.2.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.2.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.2.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.2.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.3.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.3.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.3.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.3.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.4.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.4.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.4.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.4.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.5.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.5.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.5.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.5.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.6.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.6.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.6.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.6.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.7.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.7.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.7.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.7.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.8.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.8.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.8.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.8.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.9.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.9.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.9.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.9.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.10.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.10.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.10.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.10.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.11.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.11.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.11.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.11.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.12.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.12.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.12.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.12.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.13.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.13.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.13.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.13.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.14.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.14.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.14.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.14.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.15.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.15.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.15.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.15.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.16.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.16.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.16.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.16.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.17.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.17.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.17.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.17.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.18.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.18.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.18.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.18.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.19.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.19.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.19.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.19.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.20.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.20.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.20.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.20.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.21.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.21.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.21.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.21.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.22.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.22.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.22.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.22.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.23.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.23.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.23.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.23.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.24.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.24.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.24.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.24.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.25.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.25.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.25.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.25.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.26.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.26.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.26.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.26.mlp.dense_4h_to_h.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.27.self_attention.query_key_value.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.27.self_attention.dense.SCB', '_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.27.mlp.dense_h_to_4h.SCB',
'_TransformerLightningModule__backbone.transformer.transformer.encoder.layers.27.mlp.dense_4h_to_h.SCB'])

[ssbuild]

lora_info_args = {
'with_lora': True, # 是否启用lora模块
'r': 8,
'target_modules': train_target_modules_maps[train_model_config['model_type']],
'target_dtype': None,
'lora_alpha': 32,
'lora_dropout': 0.1,
'bias': 'none', # Bias type for Lora. Can be 'none', 'all' or 'lora_only'"
'modules_to_save' : None, # "List of modules apart from LoRA layers to be set as trainable and saved in the final checkpoint. "
'layers_to_transform': None,
'layers_pattern': None,
}
lora 冻结层修改layers_to_transform ， 例如 'layers_to_transform' = [9,10,11....] 表示 9,10,11 层训练，其它层冻结。 即可正常推理。

[dayu1979]

感谢回复，我补充一下报错的问题;
没有启用lora（之前测试lora发现容易崩溃），因此改成只是finetune，因为我是单卡，
main.py:
enable_deepspeed = False #我未启用deep_speech
enable_ptv2 = False
enable_lora = False
load_in_bit = 8 # 4 load_in_4bit, 8 load_in_8bit other 0

所以应该采用的是 lsft_config（我修改为）：
global_args = {
"load_in_8bit": True, # qlora int8
"load_in_4bit": False, # qlora int4

# load_in_4bit 量化配置
"quantization_config": None,
"num_layers_freeze": 24, # 非lora,非p-tuning 模式 ， <= config.json num_layers
"pre_seq_len": None,    #p-tuning-v2 参数 , None 禁用p-tuning-v2
"prefix_projection": False, #p-tuning-v2 参数
"num_layers": -1, # 是否使用骨干网络的全部层数 最大1-28， -1 表示全层, 否则只用只用N层

}
我直接冻结前面24层，只训练后面4层，不知这个设置是否正确，但是确实在epoch0训练完成时，在best_ckpt目录下生成了一个size是7个多G的文件:last.ckpt
当我试图用infer_fintune.py进行推断的时候报错。

[ssbuild]

推理的时候 load_in_8bit = True