Handle exceptions when invoking LLM App and retrieving JSON from URI #1320
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aybruhm
requested changes
Jan 31, 2024
Comment on lines
+188
to
+200
| for _ in testset_data: | ||
| list_of_app_outputs.append( | ||
| InvokationResult( | ||
| result=Result( | ||
| type="error", | ||
| error=Error( | ||
| message="An error occurred while invoking the LLM App", | ||
| stacktrace=str(e), | ||
| ), | ||
| ) | ||
| ) | ||
| ) | ||
| return list_of_app_outputs |
There was a problem hiding this comment.
What happens in the case where a user has 50 data in their testset, evaluation runs for the first 10, and the eleventh fails?
Are we marking the status of that evaluation as partial fail? Or, failed?
If it's the former, I'd recommend we minus the length of list_of_app_outputs from the testset_data and only record the failed datapoint to the last. If it's the latter (marking the entire evaluation as failed), then we should clear the list_of_app_outputs so we don't end up having duplicate datapoint results.
See suggestion:
len_of_app_outputs = len(list_of_app_outputs)
for _ in testset_data[len_of_app_outputs:]:
# continue logic
aakrem
reviewed
Feb 5, 2024
| resp = await client.get(uri, timeout=timeout) | ||
| json_data = json.loads(resp.text) | ||
| return json_data | ||
| except Exception as e: |
There was a problem hiding this comment.
this won't be populated to the user.
aakrem
requested changes
Feb 5, 2024
| except Exception as e: | ||
| logger.traceback.print_exc() | ||
| for _ in testset_data: | ||
| list_of_app_outputs.append( |
There was a problem hiding this comment.
I don't think we need this loop
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.