我希望添加模型评测显存释放功能，单卡24G评测llama3会显存不足

[Micla-SHL]

Reminder

• I have read the README and searched the existing issues.

Reproduction

CUDA_VISIBLE_DEVICES=0 USE_MODELSCOPE_HUB=1 python src/evaluate.py --model_name_or_path LLM-Research/Meta-Llama-3-8B-Instruct --template llama3 --finetuning_type lora --task ceval --split validation --lang zh --n_shot 5 --batch_size 4

Expected behavior

我希望在/LLaMA-Factory/src/llmtuner/eval/evaluator.py def eval(self) -> None: 函数下添加

            for i in trange(
                0, len(inputs), self.eval_args.batch_size, desc="Predicting batches", position=1, leave=False
            ):
                batch_input = self.tokenizer.pad(
                    inputs[i : i + self.eval_args.batch_size], return_attention_mask=True, return_tensors="pt"
                ).to(self.model.device)
                preds = self.batch_inference(batch_input)
                outputs += preds
                del batch_input                                        #add  code
                torch.cuda.empty_cache()                       #add  code

我在评测循环尾部添加了显存释放模块，我的操作是合法的吗？ 有不利的地方吗？ 它至少能让我在单卡24G上做评测

System Info

No response

Others

我还不会做github 代码提交，所以暂时以提问形式作补充：

if 我的操作有什么更优的方式，希望能分享一下

[Naozumi520]

Do smaller batch_size help? I see you using batch size 4. I'm doing full parameter finetuning with badam so I think you should be able to do eval by lower the batch_size...

[hiyouga]

--batch_size 1

[Micla-SHL]

Do smaller batch_size help? I see you using batch size 4. I'm doing full parameter finetuning with badam so I think you should be able to do eval by lower the batch_size...

谢谢，我在刚提交该问题的时候考虑过batch_size, 它不太是我想解决的办法，所以还是想问问看，我这样纯粹的释放显存，有没有不好的地方