from_openai doesn't work with llama-cpp-python

[ahuang11]

• This is actually a bug report.
• I am not getting good LLM Results
• I have tried asking for help in the community on discord or discussions and have not received a response.
• I have tried searching the documentation and have not found an answer.

What Model are you using?

• gpt-3.5-turbo
• gpt-4-turbo
• gpt-4
• Other (please specify)

Describe the bug
A clear and concise description of what the bug is.

The documented example doesn't work
https://github.com/jxnl/instructor/blob/main/docs/hub/llama-cpp-python.md?plain=1

To Reproduce
Steps to reproduce the behavior, including code snippets of the model and the input data and openai response.

I tried to update it:

import llama_cpp
from llama_cpp.llama_speculative import LlamaPromptLookupDecoding

import instructor

from pydantic import BaseModel
from typing import List
from rich.console import Console

from huggingface_hub import hf_hub_download

model_path = hf_hub_download(
    "TheBloke/OpenHermes-2.5-Mistral-7B-GGUF",
    "openhermes-2.5-mistral-7b.Q4_K_M.gguf",
)
llama = llama_cpp.Llama(
    model_path=model_path,
    n_gpu_layers=-1,
    chat_format="chatml",
    n_ctx=2048,
    draft_model=LlamaPromptLookupDecoding(num_pred_tokens=2),  # (1)!
    logits_all=True,
    verbose=False,
)


create = instructor.from_openai(
    llama,
    mode=instructor.Mode.JSON_SCHEMA,  # (2)!
).create_chat_completion_openai_v1


text_block = """
In our recent online meeting, participants from various backgrounds joined to discuss
the upcoming tech conference. The names and contact details of the participants were as follows:

- Name: John Doe, Email: johndoe@email.com, Twitter: @TechGuru44
- Name: Jane Smith, Email: janesmith@email.com, Twitter: @DigitalDiva88
- Name: Alex Johnson, Email: alexj@email.com, Twitter: @CodeMaster2023

During the meeting, we agreed on several key points. The conference will be held on March 15th, 2024,
at the Grand Tech Arena located at 4521 Innovation Drive. Dr. Emily Johnson, a renowned AI researcher,
will be our keynote speaker.

The budget for the event is set at $50,000, covering venue costs, speaker fees, and promotional activities.
Each participant is expected to contribute an article to the conference blog by February 20th.

A follow-up meetingis scheduled for January 25th at 3 PM GMT to finalize the agenda and confirm the list of speakers.
"""


class User(BaseModel):
    name: str
    email: str
    twitter: str


class MeetingInfo(BaseModel):
    users: List[User]
    date: str
    location: str
    budget: int
    deadline: str


extraction_stream = create(
    response_model=instructor.Partial[MeetingInfo],  # (3)!
    messages=[
        {
            "role": "user",
            "content": f"Get the information about the meeting and the users {text_block}",
        },
    ],
    stream=True,
)


console = Console()

for extraction in extraction_stream:
    obj = extraction.model_dump()
    console.clear()  # (4)!
    console.print(obj)

AttributeError: 'Llama' object has no attribute 'base_url'

Expected behavior
A clear and concise description of what you expected to happen.

It should not raise an error about base_url

Screenshots
If applicable, add screenshots to help explain your problem.

[dosubot]

Hello @ahuang11! I'm Dosu, a bot here to lend a hand with bugs, answer questions, and guide you towards becoming a contributor while we wait for a human maintainer to step in. I'm diving into your issue now and will get back to you with a detailed answer shortly.

[jxnl]

Understood. Use instructor.patch for now.

And we'll need to make a PR to make sure base York exists first before we check it.

[kadirnar]

@jxnl
Can you share sample code? I'm getting the same error.

[dosubot]

To work around the AttributeError: 'Llama' object has no attribute 'base_url' issue, you can apply a temporary patch in your code. This involves manually setting the base_url attribute on the Llama object or modifying the usage to avoid accessing base_url directly. Since the maintainer suggested using an instructor patch, here's a basic example of how you might implement a workaround:

# Assuming you have your Llama object initialized as `llama`
# Directly setting the base_url attribute as a workaround
llama.base_url = "your_base_url_here"

# Proceed with your code logic...

Please replace "your_base_url_here" with the actual base URL you intend to use, if applicable. This is a temporary fix until the official patch is released. Keep an eye on the repository for updates or a permanent solution to this issue.

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[jxnl]

the base_url issue will be fixed in 1.2.3

[ahuang11]

Thanks for the update.

I think there's still an assert statement blocking this:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[6], line 28
     13 model_path = hf_hub_download(
     14     "TheBloke/OpenHermes-2.5-Mistral-7B-GGUF",
     15     "openhermes-2.5-mistral-7b.Q4_K_M.gguf",
     16 )
     17 llama = llama_cpp.Llama(
     18     model_path=model_path,
     19     n_gpu_layers=-1,
   (...)
     26     verbose=False,
     27 )
---> 28 create = instructor.from_openai(
     29     client=llama,
     30     mode=instructor.Mode.JSON_SCHEMA,  # (2)!
     31 )
     32 message = {"role": "user", "content": "Teach me how to say `Hello` in three languages!"}
     33 extraction_stream = create(
     34     response_model=instructor.Partial[Translations],  # (3)!
     35     messages=[message],
     36     stream=True,
     37 )

File ~/miniconda3/envs/panel/lib/python3.10/site-packages/instructor/client.py:277, in from_openai(client, mode, **kwargs)
    274 else:
    275     provider = Provider.OPENAI
--> 277 assert isinstance(
    278     client, (openai.OpenAI, openai.AsyncOpenAI)
    279 ), "Client must be an instance of openai.OpenAI or openai.AsyncOpenAI"
    281 if provider in {Provider.ANYSCALE, Provider.TOGETHER}:
    282     assert mode in {
    283         instructor.Mode.TOOLS,
    284         instructor.Mode.JSON,
    285         instructor.Mode.JSON_SCHEMA,
    286         instructor.Mode.MD_JSON,
    287     }

AssertionError: Client must be an instance of openai.OpenAI or openai.AsyncOpenAI

[jxnl]

im dumb im so sorry.

[jxnl]

i need to add tests for this.

[amaarora]

Thank you! I am trying to use llama with Instructor as well to get structured outputs. Are you suggesting that using instructor.patch is the best way to go forward at the moment?

[guang]

@amaarora if ur running llama on groq, this worked for me

groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
client = instructor.from_groq(groq_client, mode=instructor.Mode.JSON)
chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content":sys_prompt,
        },
        {
            "role": "user",
            "content":user_prompt,
        },
    ],
    model="llama3-70b-8192",
    temperature=temp,
    max_retries=Retrying(
        stop=stop_after_attempt(2),
        wait=wait_fixed(60), # rate limit on groq lol
    ),
    response_model=self.response_model,
)