New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A/B Test LLMs? #56

Open

krrishdholakia opened this issue Sep 3, 2023 · 0 comments

krrishdholakia commented Sep 3, 2023 •

edited

In production, we don't know if Llama2 is going to provide:

good results
quickly

Would it be helpful to provide a way to easily A/B test between new models in production?

Context - I'm working on LiteLLM and we recently released a way to a/b test straight from the completion endpoint:

Tutorial: https://docs.litellm.ai/docs/tutorials/ab_test_llms

Dashboard:

krrishdholakia changed the title ~~A/N T~~ A/B Test LLMs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment