Fix exception handling gemini #3493

nobu007 · 2024-05-07T00:56:34Z

Fix exception handling gemini

Add get_provider_from_model() for handling 429 Exception from gemini.

Relevant issues

This error is the target problem to resolved in PR.
From some case, model is not include llm_provider name.
It causes ERROR in get_llm_provider().

"""
09:26:06 - LiteLLM:ERROR: utils.py:5873 - Error occurred in getting api base - GetLLMProvider Exception - name 'get_provider_from_model' is not defined

original model: gemini-1.5-pro-latest
"""

Type

🧹 Refactoring

Changes

Add get_provider_from_model()
Call get_provider_from_model() from get_llm_provider() before exception.
The dict provider_by_model manages full model(provider/model_name ) and model_name only.

Testing

Works fine with this condition.

MODEL_NAME="gemini/gemini-1.5-pro-latest"
GEMINI_API_KEY=XXXX
LITELLM_LOCAL_MODEL_COST_MAP="True"

Finaly

Error in chat: 429 Resource has been exhausted (e.g. check quota).

Notes

Retlated with #3473.
This PR may help to reproduce the condition.
"gemini/gemini-1.5-pro-latest" soon 429 exception.

Pre-Submission Checklist (optional but appreciated):

I have included relevant documentation updates (stored in /docs/my-website)

OS Tests (optional but appreciated):

Tested on Windows
Tested on MacOS
Tested on Linux

vercel · 2024-05-07T00:57:44Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 7, 2024 2:33pm

krrishdholakia · 2024-05-07T04:52:09Z

litellm/utils.py

 from .integrations.prometheus import PrometheusLogger
 from .integrations.prometheus_services import PrometheusServicesLogger
-from .integrations.dynamodb import DyanmoDBLogger
+from .integrations.prompt_layer import PromptLayerLogger


is there a reason why so many imports in here+ init.py are refactored?

It makes it hard to isolate the actual changes you're trying to make in this PR

It is just changed by isort.
If you not want to aplly, I will drop "style(utils.py): apply formatter" commit.

yes, please. it would help isolate the changes in this PR

Thank you for your work on this!

I dropped it.
Sorry, my commit include it.
Wait a moment please.

I fixed the many imports change.

krrishdholakia · 2024-05-07T04:53:32Z

litellm/utils.py

@@ -4602,6 +4606,9 @@ def register_model(model_cost: Union[str, dict]):
        elif value.get("litellm_provider") == "bedrock":
            if key not in litellm.bedrock_models:
                litellm.bedrock_models.append(key)
+        elif value.get("litellm_provider") == "gemini":
+            if key not in litellm.gemini_models:
+                litellm.gemini_models.append(key)


Is this needed? All gemini models start with gemini/ and so already work without changes.

Can you share an example of the problem you're trying to solve with this PR?

In my invessgation, this code calls model without "gemini/" in case some Exception.

This is stack trace.

09:26:06 - LiteLLM:ERROR: utils.py:5873 - Error occurred in getting api base - GetLLMProvider Exception - name 'get_provider_from_model' is not defined original model: gemini-1.5-pro-latest fixed_litellm_completions second_error e2= 400 API key not valid. Please pass a valid API key. [reason: "API_KEY_INVALID" domain: "googleapis.com" metadata { key: "service" value: "generativelanguage.googleapis.com" } ] Error in chat: 429 Resource has been exhausted (e.g. check quota). Traceback (most recent call last): File "/app/litellm/litellm/llms/gemini.py", line 213, in completion response = _model.generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/generativeai/generative_models.py", line 256, in generate_content iterator = self._client.stream_generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 1090, in stream_generate_content response = rpc( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ return wrapped_func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func return retry_target( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target _retry_error_helper( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target result = target() File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout return func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 174, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.InvalidArgument: 400 API key not valid. Please pass a valid API key. [reason: "API_KEY_INVALID" domain: "googleapis.com" metadata { key: "service" value: "generativelanguage.googleapis.com" } ] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/litellm/litellm/main.py", line 1684, in completion model_response = gemini.completion( File "/app/litellm/litellm/llms/gemini.py", line 241, in completion raise GeminiError( litellm.llms.gemini.GeminiError: 400 API key not valid. Please pass a valid API key. [reason: "API_KEY_INVALID"

i'm confused - yes, we make calls to gemini without the gemini/ as that is used by litellm as internal logic for routing.

Where is the get_llm_provider error in this stack trace?

First of all, "LiteLLM:ERROR: utils.py:5873 - Error occurred in getting api base " shows the exception from get_llm_provider
here (line number is little changed by adding log).

This is stack trace from get_llm_provider().
It looks some times Exception handled.
I think the Exceition should be handked more early step.
This PR can fix "LiteLLM:ERROR" in get_llm_provider().

During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/litellm/litellm/utils.py", line 6391, in get_llm_provider raise litellm.exceptions.BadRequestError( # type: ignore litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=gemini-1.5-pro-latest

This is full stack trace I get for one ERROR.
The "LiteLLM:ERROR" is show 2 times in real.
This behaivior was a little noisy to debug for me.

params_dump= {'model': 'gemini/gemini-1.5-pro-latest', 'messages': [{'role': 'system', 'content': 'You are Open Interpr'}, {'role': user, 'content': 'pythonで円周率を表示するプログラム'}, {'role': assistant, 'content': ''}, {'role': user, 'content': '会話履歴で状況を確認してから自動的に処理'}], 'stream': True, 'max_tokens': 4096} Traceback (most recent call last): File "/app/litellm/litellm/llms/gemini.py", line 213, in completion response = _model.generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/generativeai/generative_models.py", line 256, in generate_content iterator = self._client.stream_generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 1090, in stream_generate_content response = rpc( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ return wrapped_func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func return retry_target( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target _retry_error_helper( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target result = target() File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout return func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 174, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.ResourceExhausted: 429 Resource has been exhausted (e.g. check quota). During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/litellm/litellm/main.py", line 1684, in completion model_response = gemini.completion( File "/app/litellm/litellm/llms/gemini.py", line 241, in completion raise GeminiError( litellm.llms.gemini.GeminiError: 429 Resource has been exhausted (e.g. check quota). During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/litellm/litellm/utils.py", line 6391, in get_llm_provider raise litellm.exceptions.BadRequestError( # type: ignore litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=gemini-1.5-pro-latest Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers 12:07:08 - LiteLLM:ERROR: utils.py:5870 - Error occurred in getting api base - LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=gemini-1.5-pro-latest Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers fixed_litellm_completions the first_error e= 429 Resource has been exhausted (e.g. check quota). Traceback (most recent call last): File "/app/litellm/litellm/llms/gemini.py", line 213, in completion response = _model.generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/generativeai/generative_models.py", line 256, in generate_content iterator = self._client.stream_generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 1090, in stream_generate_content response = rpc( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ return wrapped_func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func return retry_target( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target _retry_error_helper( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target result = target() File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout return func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 174, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.InvalidArgument: 400 API key not valid. Please pass a valid API key. [reason: "API_KEY_INVALID" domain: "googleapis.com" metadata { key: "service" value: "generativelanguage.googleapis.com" } ] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/litellm/litellm/main.py", line 1684, in completion model_response = gemini.completion( File "/app/litellm/litellm/llms/gemini.py", line 241, in completion raise GeminiError( litellm.llms.gemini.GeminiError: 400 API key not valid. Please pass a valid API key. [reason: "API_KEY_INVALID" domain: "googleapis.com" metadata { key: "service" value: "generativelanguage.googleapis.com" } ] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/litellm/litellm/utils.py", line 6391, in get_llm_provider raise litellm.exceptions.BadRequestError( # type: ignore litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=gemini-1.5-pro-latest Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers 12:07:08 - LiteLLM:ERROR: utils.py:5870 - Error occurred in getting api base - LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=gemini-1.5-pro-latest Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers fixed_litellm_completions second_error e2= 400 API key not valid. Please pass a valid API key. [reason: "API_KEY_INVALID" domain: "googleapis.com" metadata { key: "service" value: "generativelanguage.googleapis.com" } ] Error in chat: 429 Resource has been exhausted (e.g. check quota). Traceback (most recent call last): File "/app/litellm/litellm/llms/gemini.py", line 213, in completion response = _model.generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/generativeai/generative_models.py", line 256, in generate_content iterator = self._client.stream_generate_content( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 1090, in stream_generate_content response = rpc( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ return wrapped_func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func return retry_target( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target _retry_error_helper( File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target result = target() File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout return func(*args, **kwargs) File "/home/jinno/.pyenv/versions/3.9.19/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 174, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.InvalidArgument: 400 API key not valid. Please pass a valid API key. [reason: "API_KEY_INVALID" domain: "googleapis.com" metadata { key: "service" value: "generativelanguage.googleapis.com" } ]

krrishdholakia · 2024-05-09T05:17:16Z

@nobu007 can you share a sample code snippet to repro the issue, this PR tries to solve?

nobu007 · 2024-05-12T04:00:44Z

@krrishdholakia
This is sample snippet.

import litellm
from dotenv import load_dotenv

load_dotenv(verbose=True, override=False)

params = {
    "model": "gemini/gemini-1.5-pro-latest",
    "messages": [
        {"role": "system", "content": "You are good programmer."},
        {"role": "user", "content": "Please write a python proram that show Pi"},
    ],
    "stream": True,
    "max_tokens": 4096,
}


def test():
    for i in range(10):
        print("i=", i)
        litellm.completion(**params)


test()

This is result of my environment.

(gamebook) jinno@jinno-desktop:~/git/drill/gamebook$ python test/test_litellm.py 
Traceback (most recent call last):
  File "/home/jinno/git/drill/gamebook/test/test_litellm.py", line 10, in <module>
    {"role": user, "content": "pythonで円周率を表示するプログラム"},
NameError: name 'user' is not defined
(gamebook) jinno@jinno-desktop:~/git/drill/gamebook$ python test/test_litellm.py 
i= 0
i= 1
i= 2
i= 3
i= 4
i= 5

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.


Provider List: https://docs.litellm.ai/docs/providers

Traceback (most recent call last):
  File "/home/jinno/git/drill/gamebook/open_interpreter_docker/litellm/litellm/llms/gemini.py", line 213, in completion
    response = _model.generate_content(
・・・
litellm.exceptions.APIConnectionError: 429 Resource has been exhausted (e.g. check quota).

nobu007 · 2024-05-12T04:05:10Z

Please use for testing "gemini/gemini-1.5-pro-latest". It is easy to repro 429 error.
Of course, similar problems may occur with other models depending on the timing.
#3473

krrishdholakia · 2024-05-12T04:13:22Z

@nobu007

429 Resource has been exhausted (e.g. check quota).

Your error isn't being caused by litellm though - that just looks like you ran out of your vertex ai quota.

What is the litellm error here?

nobu007 · 2024-05-12T04:31:54Z

@krrishdholakia
I think litellm have not handle the Exception from gemini smart now.
These messages confuse user. Is this reasnable in this sutuation?

"LiteLLM:ERROR: utils.py:5870 - Error occurred in getting api base - LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=gemini-1.5-pro-latest"

"litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=gemini-1.5-pro-latest"

nobu007 mentioned this pull request May 7, 2024

Add gemini models for litellm/__init__.py #3473

Open

4 tasks

vercel bot deployed to Preview May 7, 2024 00:58 View deployment

krrishdholakia reviewed May 7, 2024

View reviewed changes

nobu007 force-pushed the fix_exception_handling_gemini branch from 0cd7d64 to 16fa8a7 Compare May 7, 2024 14:27

nobu007 marked this pull request as ready for review May 7, 2024 14:27

vercel bot deployed to Preview May 7, 2024 14:27 View deployment

nobu007 added 2 commits May 7, 2024 23:32

fix(litellm/__init__.py): add provider_by_model dict

9e6924a

fix(utils.py): add get_provider_from_model()

47e98cc

nobu007 force-pushed the fix_exception_handling_gemini branch from 16fa8a7 to 47e98cc Compare May 7, 2024 14:33

vercel bot deployed to Preview May 7, 2024 14:33 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix exception handling gemini #3493

Fix exception handling gemini #3493

nobu007 commented May 7, 2024 •

edited

vercel bot commented May 7, 2024 •

edited

krrishdholakia May 7, 2024

nobu007 May 7, 2024 •

edited

krrishdholakia May 7, 2024 •

edited

nobu007 May 7, 2024 •

edited

nobu007 May 7, 2024

krrishdholakia May 7, 2024

nobu007 May 7, 2024

krrishdholakia May 9, 2024

nobu007 May 12, 2024

nobu007 May 12, 2024

krrishdholakia commented May 9, 2024

nobu007 commented May 12, 2024

nobu007 commented May 12, 2024

krrishdholakia commented May 12, 2024

nobu007 commented May 12, 2024

Fix exception handling gemini #3493

Are you sure you want to change the base?

Fix exception handling gemini #3493

Conversation

nobu007 commented May 7, 2024 • edited