-
Notifications
You must be signed in to change notification settings - Fork 129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evaluation settings of INSTRUCTOR #67
Comments
Hi, the instructions are included in the evaluation. You may refer to the table 1 in our paper |
As far as I understand, when evaluating MTEB in your code, the following lines are used: model = INSTRUCTOR(args.model_name, cache_folder=args.cache_dir)
evaluation = MTEB(tasks=[args.task_name], task_langs=["en"])
evaluation.run(model, output_folder=args.output_dir, eval_splits=[args.split], args=args, overwrite_results=True) During the execution of I'm not sure if my understanding and evaluation method are correct. I would greatly appreciate it if you could provide me with answers. Thank you very much. |
Hi, could you share the scripts you print out the sentences? Also, make sure you have correctly installed the InstructorEmbedding library. |
I just use the source code on the Github: https://github.com/HKUNLP/instructor-embedding/blob/main/InstructorEmbedding/instructor.py#L478-L565
I print |
For the case of MTEB, make sure that the library is correctly installed by following https://github.com/HKUNLP/instructor-embedding#mteb. |
Sorry, I have tried to install this previously but failed with the message here:
So I can only Maybe this is the reason for the problem? |
Yes, we should install the customized mteb package for correct evaluation. |
Thanks! I've corrected my evaluation method following your customized mteb package. The performance of replicating INSTRUCTOR have been improved but still lower than yours. Here I still have some detailed questions:
Thank you again for your patient response. |
Yes, we use the mean pooling in both the training and evaluation processes. |
I guess the reason for this issue might be due to the exit() statement present at https://github.com/HKUNLP/instructor-embedding/blob/main/evaluation/MTEB/setup.py#L42 |
Hi, you may check the permission of /tmp or /tmp/tmp73vjuhzp. |
Hello! I have a very puzzling question that I would like to ask. Since your model is fine-tuned with instructions, why not use instructions during benchmark evaluations (e.g. MTEB)?
The text was updated successfully, but these errors were encountered: