You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bug / Assistance Description
The results that are reported in the HH column are very different to the ReAct paper. In particular, ReAct reports
To Reproduce
See screenshots below. Your results in HH column indicate 16% success for text-davinci-002 or gpt-3.5-turbo. However, the reults using text-davinci-002 on ReAct indicate 78% (second screenshot). This is a significant difference.
Screenshots or Terminal Copy&Paste
Concrete Questions / Actions:
Please tell us:
How your evaluation for Alfworld (HH) differs from ReAct?
Which exact model you used?
Which prompts you used (1-shot, 2-shot), and are they the same as from the ReAct paper?
Why are the results so different?
The text was updated successfully, but these errors were encountered:
Please read the paper carefully. You can find all the prompt in appendix or code. The results are different because 1. we are not using the same prompt. 2. we are not using exactly the same envrionment.
Can you elaborate how the environment is not exactly the same? [Do you use a different version of alfworld, etc.?]
The reason for asking about this question is to understand whether you were able to get close to the results reported in ReAct and what the exact difference might be, as the results of ReAct seem quite impossible to reproduce.
Hi, @ai-nikolai sorry for the late reply, we've been quite busy lately. To answer your question, I believe the main difference is the prompting technique. We weren't aiming to reproduce the ReAct's result, but to design a prompt and a evaluation process that is relatively fair to all the models. The prompt we used is listed in paper Appendix G. The evaluation process is located at
Bug / Assistance Description
The results that are reported in the HH column are very different to the ReAct paper. In particular, ReAct reports
To Reproduce
See screenshots below. Your results in HH column indicate 16% success for text-davinci-002 or gpt-3.5-turbo. However, the reults using text-davinci-002 on ReAct indicate 78% (second screenshot). This is a significant difference.
Screenshots or Terminal Copy&Paste
Concrete Questions / Actions:
Please tell us:
The text was updated successfully, but these errors were encountered: