Langchain::Assistant support for Google Gemini LLM #513
Conversation
andreibondarev
changed the title
There was a problem hiding this comment.
Different Message types look good to me. Then I think you can put extract_openai_tool_call and extract_gemini_tool_call into them as extract_tool_call to avoid the conditionals in assistant.rb. (Later have extract_tool_calls for mutiple/parallel tools.)
And tool_role to avoid that conditional too, but looks like you're in the middle of doing that.
P.S. Seems you could put it all in OpenAIResponse and GeminiResponse, unless you want to limit it to Assistants.
|
@andreibondarev Totally up to you, but if you merge my weather tool improvement (#602) and database tool improvement (#591) into |
|
Just pinging the thread on the status. Seems like only tests are missing? |
|
Note to self: OpenAI support more JSON Schema options (like "examples" and "default" keys) compared to Gemini. I'm sure it'll change in the future. |
| "language": { | ||
| "type": "string", | ||
| "description": "The 2-letter ISO-639-1 code of the language you want to get headlines for.", | ||
| "enum": ["ae", "ar", "at", "au", "be", "bg", "br", "ca", "ch", "cn", "co", "cu", "cz", "de", "eg", "fr", "gb", "gr", "hk", "hu", "id", "ie", "il", "in", "it", "jp", "kr", "lt", "lv", "ma", "mx", "my", "ng", "nl", "no", "nz", "ph", "pl", "pt"] |
There was a problem hiding this comment.
Missing "en" for English, and the table I found says "eg" is Unassigned.
There was a problem hiding this comment.
I think you mistakenly put the parameters for get_top_headlines country here. They should go below under news_retriever__get_top_headlines, and the language values listed in the docs should go here (https://newsapi.org/docs/endpoints/everything).
There was a problem hiding this comment.
@mattlindsey Nice find!
| country: nil, | ||
| category: nil, | ||
| sources: nil, | ||
| q: nil, |
There was a problem hiding this comment.
You probably know, but missing some parameters like 'language', 'from', 'to',...
So currently if you ask, for example, for "Top headlines from Germany in English" the tool calls fails with unknown keyword: :language (ArgumentError) and leaves the Assistant in a bad state (missing role=tool messages) and every subsequent attempt fails like below (with OpenAI):
3.2.1 :014 > assistant.add_message_and_run content:"Now give the news in Germany?", auto_tool_execution: true
I, [2024-05-06T20:49:45.668612 #8222] INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::OpenAI
OpenAI HTTP Error (spotted in ruby-openai 6.4.0): {"error"=>{"message"=>"An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_u4qPhNeJInxgjIgAQQ32AuE2", "type"=>"invalid_request_error", "param"=>"messages.[3].role", "code"=>nil}}
There was a problem hiding this comment.
I added language and en to parameter enum, but unfortunately it doesn't seem to like translating headlines to English:
"I couldn't find any latest news from Germany in English. Would you like me to search for news from a different country or on a different topic?"
Details:
"name"=>"news_retriever__get_top_headlines", "arguments"=>"{\"country\":\"de\",\"language\":\"en\",\"page_size\":5}" causes news api to return {\"status\":\"ok\",\"totalResults\":0,\"articles\":[]})
I also saw OpenAI sometimes not pass the ':language' parameter to the tool and try multiple times since the headlines returned weren't English. Wierd.
There was a problem hiding this comment.
Actually your code looks fine. Sorry. I think you have all of the correct parameters.
Although it does seem the llm tries to pass language to top_headlines even though it's not listed in allowed params. Try it: "Top headlines from Germany in English".
lib/langchain/llm/google_gemini.rb
Outdated
| headers: {"Content-Type" => "application/json"} | ||
| ) | ||
|
|
||
| if response.code != 200 |
There was a problem hiding this comment.
Maybe you need to say if response.code != 200 || response.body.empty? (or maybe if response.body == "{}\n") since we're still getting that occasionally (for GoogleGemini), which causes exception later when you try to parse it.
|
@palladius Could you please test it with GoogleVertexAI LLM? llm = Langchain::LLM::GoogleVertexAI.new(project_id: ENV["GOOGLE_GEMINI_API_KEY"], region: "us-central-1")
thread = Langchain::Thread.new
assistant = Langchain::Assistant.new(
llm: llm,
thread: thread,
instructions: "You are a News Assistant.",
tools: [Langchain::Tool::NewsRetriever.new(api_key: ENV["NEWS_API_KEY"])]
)
assistant.add_message_and_run content:"What are the latest news from Italy?", auto_tool_execution: trueDon't worry about the specs failing, I'll fix them. |
|
Hi, I got it to work with my personal account. Occasionally I get an empty/nil return: |
I'm pretty confident that this happens when the quota is exceeded. I get this occasionally as well. |
|
Awesome progress so far! Thanks for the hard work Andrei! |
|
works liek a charm for me! |
Langchain::Assistantworks withLangchain::LLM::GoogleGemininow. This PR includes a number of breaking changes:Langchain::Messageclass intoLangchain::Messages::OpenAIMessageandLangchain::Messages::GoogleGeminiMessageThe
Langchain::Assistantcan be instantiate to use Google Gemini the following way, e.g.:To add a message:
To run the assistant
To run the assistant and auto execute tools