Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re run evaluation #1455

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Re run evaluation #1455

wants to merge 28 commits into from

Conversation

aakrem
Copy link
Collaborator

@aakrem aakrem commented Mar 21, 2024

Rerunning an existing evaluation

Copy link
Member

@mmabrouk mmabrouk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @aakrem

Here are my notes:

  • I have tried deploying the branch but it did not work (see screenshot)
Screenshot 2024-03-27 at 21 56 49 - Can you please link the issue for this PR. I could not find it. I was under the impression you were working on 1396 . I don't understand the use case exactly for this PR. I think it would result in a wrong behavior (see comments about variant ids). - Overall I feel that you are overcomplicating the solution. I don't understand at for instance why we would need a rerun count.

@aakrem
Copy link
Collaborator Author

aakrem commented Mar 28, 2024

Thanks for the PR @aakrem

Here are my notes:

  • I have tried deploying the branch but it did not work (see screenshot)

Screenshot 2024-03-27 at 21 56 49 - Can you please link the issue for this PR. I could not find it. I was under the impression you were working on #1396 . I don't understand the use case exactly for this PR. I think it would result in a wrong behavior (see comments about variant ids). - Overall I feel that you are overcomplicating the solution. I don't understand at for instance why we would need a rerun count.

I couldn't reproduce that error even when accessing the same app! I however see another issue that most probably related to an evaluator created from another branch or so.
The best way to test it is to deploy main, create evals, deploy rerun, and then test it.
Check the rerun app

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants