You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It is fairly easy to create a multiple choice question where the model can reasonably want to not select a given option, but this question type throws an error if an option is not selected.
We should consider handling 'None' as we do with checkbox questions (allow it) or provide a clearer error message alerting the user to the issue.
Here I test valid "None" responses:
from edsl.questions import QuestionList, QuestionCheckBox, QuestionMultipleChoice, QuestionFreeText
from edsl import Scenario, Survey
q1 = QuestionList(
question_name = "list",
question_text = "List the countries mentioned in this text: {{ text }}"
)
q2 = QuestionCheckBox(
question_name = "checkbox",
question_text = "Select all the countries mentioned in this text: {{ text }}",
question_options = ["United States", "Canada", "Mexico"]
)
q3 = QuestionMultipleChoice(
question_name = "choice",
question_text = "Select the first country mentioned in this text: {{ text }}",
question_options = ["United States", "Canada", "Mexico"]
)
q4 = QuestionFreeText(
question_name = "free",
question_text = "Identify the countries mentioned in this text: {{ text }}"
)
texts = ["This is how you should brush your teeth.", "This is how to make coffee."]
scenarios = [Scenario({"text":t}) for t in texts]
survey = Survey([q1,q2,q3,q4])
The errors are only with QuestionMultipleChoice:
The text was updated successfully, but these errors were encountered:
Maybe? We don't want to get into validating all "None"-like responses, and then having to decide what we translate them to in results for consistency. We could try pairing it with an additional instruction to return "None" but I think that actually makes it less consistent with checkbox where we allow specified min/max, and I think consistency between these types is important.
I think my first preference is to leave it as is but add a clear error message that the model did not select a response for the multiple choice question. This has the benefit of alerting the user.
We could do the same for checkbox questions where the min/max have been specified but the model has not strayed.
I do think it's a situation where we do not necessarily want to mirror human surveys, where you can be prevented from moving on if you don't answer. It's arguably an improvement that we allow the model to implicitly point out that the question is bad, instead of forcing a non-sensible response to it.
It is fairly easy to create a multiple choice question where the model can reasonably want to not select a given option, but this question type throws an error if an option is not selected.
We should consider handling 'None' as we do with checkbox questions (allow it) or provide a clearer error message alerting the user to the issue.
Here I test valid "None" responses:
The errors are only with QuestionMultipleChoice:
The text was updated successfully, but these errors were encountered: