Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

79 Open 255 Closed
79 Open 255 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[Feature] 8卡A100*80G评测Qwen1.5-72B推荐的参数设置是怎么样的呢
#1135 opened May 10, 2024 by boundles
1 task
@tonysy
/home/wumao/opencompass-main/opencompass/summarizers/needlebench.py 的sizes是不是少了一个128k [Bug] [Feature]
#1132 opened May 10, 2024 by 1518630367
1 task done
@tonysy
官网上每个数据集所使用的config文件的版本如何知道？
#1130 opened May 9, 2024 by jyshee
@kennymckormick
1
新老版本的结果差距
#1126 opened May 9, 2024 by mengrusun
2 tasks done
@kennymckormick
1
[Bug] Base classes use @abstractmethod without subclassing from abc.ABC
#1124 opened May 8, 2024 by acylam
2 tasks done
@Leymore @liushz
[Improvement] Mutiprocess Evaluation time Bug
#1115 opened May 6, 2024 by wdndev
2 tasks done
@Leymore
1
[Feature] 请问在使用VLLM测评模型humaneval时，batch_size 不同导致 测评结果有区别是为什么？
#1097 opened Apr 26, 2024 by noforit
1 task
@tonysy
[Feature] ollama发布的模型如何评估支持
#1095 opened Apr 26, 2024 by 18600709862
1 task
@tonysy
1
[Bug] 不支持flores200数据集评测
#1083 opened Apr 24, 2024 by Liukairong2023
2 tasks done
@liushz
4
[Feature] 是否可以对arc-e和arc-c的base模型评估增加25-shot with choice
#1082 opened Apr 24, 2024 by siftxxx
1 task
@liushz
[Bug] There is a large gap in the partial subset of the long text evaluation data set
#1061 opened Apr 19, 2024 by bullw
2 tasks done
@tonysy
[Bug] DLCRunner无法获取运行结果
#1058 opened Apr 18, 2024 by Ethan-9606
2 tasks done
@kennymckormick
[Feature]在model文件中没有找到关于Qwen1.5-32b-chat的配置文件
#1057 opened Apr 18, 2024 by Egber1t
1 task
@kennymckormick
2
[Feature] 兼容torch_npu
#1045 opened Apr 12, 2024 by li126com
1 task
@tonysy
1
[Feature] 自定义数据集命令如何修改评估指标？
#1044 opened Apr 12, 2024 by liushiton
1 task
@tonysy
使用baichuan-7b评估humaneval数据与榜单差距过大
#1038 opened Apr 11, 2024 by zh190920
2 tasks done
@kennymckormick
2
为什么基于openai api部署的llama2-7b-chat-hf，在MMLU数据集上测试精度远低于官方数据
#1037 opened Apr 11, 2024 by wz0424
2 tasks done
@kennymckormick
7
[Bug] Long text evaluation parameters are not clear
#1035 opened Apr 10, 2024 by bullw
2 tasks done
@liushz
3
在测评的时候显存总是有空闲，如何全部利用显存呢，单机 8*80G的 A800
#1017 opened Apr 2, 2024 by listwebit
2 tasks done
@Leymore @bittersweet1999
6
[Feature] Support PromptCBLUE
#1012 opened Mar 29, 2024 by TaoSunVoyage
1 task
@tonysy
1
[Bug] Error when evaluate using LightLLM api
#1008 opened Mar 28, 2024 by darrenglow
2 tasks done
@kennymckormick
1
[Bug] AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
#1007 opened Mar 28, 2024 by A-A-T-i-P
2 tasks done
@kennymckormick
1
[Feature] Add documentation and example for NumWorkersPartitioner
#1005 opened Mar 27, 2024 by tonysy
1 task
@Leymore
[Feature] 请问使用vllm评测时怎么实现类似HF多卡数据并行？
#1002 opened Mar 26, 2024 by noforit
1 task
@tonysy @bittersweet1999
15
[Feature] Support LiveCodeBench
#980 opened Mar 18, 2024 by Ezra-Yu
1 task
@Leymore
2
ProTip! What’s not been updated in a month: updated:<2024-04-10.