Instructor with llmlite doenst support vision for Claude 3 Haiku and Sonnet models

[enoch3712]

• This is actually a bug report.
• I am not getting good LLM Results
• I have tried asking for help in the community on discord or discussions and have not received a response.
• I have tried searching the documentation and have not found an answer.

What Model are you using?

• gpt-3.5-turbo
• gpt-4-turbo
• gpt-4
• Other (Claude 3 vision)

Describe the bug
Instructor with llmlite doesn't support vision for Claude 3 Haiku and Sonnet models

Information about the models is here:
https://docs.litellm.ai/docs/providers/anthropic

#claude-3-opus-20240229 - ok
#claude-3-haiku-20240307 - not ok
#claude-3-sonnet-20240229 - not ok

To Reproduce

# change model to "claude-3-haiku-20240307" or "claude-3-sonnet-20240229"
 resp = instructor.from_litellm(litellm.completion).chat.completions.create(
                model='claude-3-opus-20240229',
                messages=[
                    {
                        "role": "user",
                        "content": [
                            {
                                "type": "text",
                                "text": "What’s in this image?"
                            },
                            {
                                "type": "image_url",
                                "image_url": {
                                    "url": "data:image/jpeg;base64," + encode_image(file)
                                }
                            }
                        ]
                    }
                ],
                response_model=response_model,
            )

Expected behavior
Should return a response, instead raises the exception on the screenshot

Screenshots

[dosubot]

It appears the issue you're encountering with the Claude 3 Haiku and Sonnet models in the Instructor repository for vision tasks is due to an assertion failure, suggesting that the "Instructor" object does not support multiple tool calls. The error recommends using "list[Model]" instead, indicating a problem with how the tool calls are structured in your code.

To address this, ensure your implementation aligns with the repository's guidelines for handling vision tasks, particularly with models like Claude 3 Haiku and Sonnet. The repository's documentation and examples, such as examples/vision/run.py, might offer insights into the expected structure and usage patterns, though they primarily showcase GPT-4 vision model integration.

If the documentation does not explicitly cover the Claude 3 Haiku and Sonnet models for vision tasks, this might indicate a gap in the current repository support or documentation. Considering there's an ongoing discussion in the repository's issues section regarding this matter, it would be beneficial to monitor that for any updates or solutions shared by the community or maintainers.

For a more immediate resolution, providing the exact error message or exception details could help in diagnosing the issue more accurately. Additionally, sharing more about your implementation approach with the Claude 3 models could clarify the problem and facilitate targeted assistance.

Details

Sources

• Instructor with llmlite doenst support vision for Claude 3 Haiku and Sonnet models
• examples/vision/run.py
• README.md

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[ZhymabekRoman]

Workaround for now is to use two Anthropic's instances, one is patched with instructor another without.

[enoch3712]

Workaround for now is to use two Anthropic's instances, one is patched with instructor another without.

Thank you.

Is this something that can be solved easily?

Or is an known issue that you persist?