New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] 自定义数据集命令如何修改评估指标？ #1044

Open

1 task

liushiton opened this issue Apr 12, 2024 · 0 comments

Open

1 task

[Feature] 自定义数据集命令如何修改评估指标？ #1044

liushiton opened this issue Apr 12, 2024 · 0 comments

Assignees

liushiton commented Apr 12, 2024

Describe the feature

python run.py
--models hf_llama2_7b
--custom-dataset-path xxx/test_qa.jsonl
--custom-dataset-data-type qa
--custom-dataset-infer-method gen

使用这个命令得到的结果得分默认是accuracy。这意味着要完全相同才能算对么？如何替换成别的评估指标呢？
通过新增配置文件，学习成本有点高。。。

Will you implement it?

I would like to implement this feature and create a PR!

mm-assistant bot assigned tonysy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment