"You uploaded an unsupported image" when trying to streamText with image data #1544
Comments
|
The error comes from the OpenAI API. Can you test whether you also receive it when using the OpenAI fetch API directly? |
|
When I use the OpenAI SDK directly (as shown below) - I get a valid response from OpenAI. Note - I need to change the Using Sent message (using OpenAI message schema) const messages = [
{
role: 'user',
content: [
{
type: 'text',
text: 'What is in this image',
},
{
type: 'image_url',
image_url: {
url: 'https://static01.nyt.com/images/2024/05/12/.....',
},
},
],
}
];Route handler import OpenAI from "openai";
export async function POST(req: Request) {
const { messages } = await req.json();
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
const response = await openai.chat.completions.create({
model = 'gpt-4-turbo',
messages,
});
console.log(response.choices[0]);
}I feel like I must be missing something very obvious if no one else sees this behavior - but I also don't see an example in the AI SDK for messages with image content. My |
|
Maybe worth also noting is that client-side I am using the |
|
This is related to #1555 because when I remove the However, when I pass a URL I still get an error. So part of the fix may be just a matter of providing better documentation for messages with image content, but there's still something funky when sending an image URL. |
|
@lucasishuman can you share the img? |
|
You can use any valid image URL to test, such as... const result = await streamText({
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'What’s in this image?',
},
{
type: 'image',
image:
'https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg',
},
],
},
],
model: openai('gpt-4-turbo'),
}); |
|
@lucasishuman try using a URL image: new URL('https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg')See examples/ai-core/src/generate-text/openai-multimodal-url.ts for a full example |
|
@lgrammel - Great, thanks. That was the missing info I needed. I feel like adding the image data and image URL examples to the AI SDK docs would be helpful. You can close this issue. |
Description
When I attempt to stream a text response from OpenAI and image data is present in the outgoing messages, I get an error. (see below for code sample)
Model is
gpt-4-turboI'm confident it is a valid image format and under size limit.
I see the same behavior whether the image content is a URL or base64 encoded string.
Error:
I'm using:
Code example
Example user messages with image data included
Local API route
Server returns error
Additional context
No response
The text was updated successfully, but these errors were encountered: