Assistants API Support

[Phodaie]

What are your thoughts on supporting Assistants API? parallel function calls are one advantage.
I feel the API may change over the next few months, so maybe we should wait.

[jxnl]

no goals for assistant, my money is on marvin ai doing a great job

[PaulGilmartin]

I would also love for the Assistants API to be supported

[jxnl]

what is the application you're looking for?

[Phodaie]

Developing full application is much easier with Assistants API as OpenAI does all the state-management for you. The application just need to keep the entity ids: Assiatand_id, Thread_id, run_id. Also all entities support "metadata" that can be used to store more application specific and situational attributes. This is perfect of server-less apps.
Also it seems that OpenAI's focus is on this so it make sense for us to put all of our API related effort on it. I think other LLM instance providers will follow OpenAI direction.

I am working on a multi-agent system. Each agent is an Assistant. These Assistants are used in the same Thread. So I can switch Assistant based on the current state of the Thread.

[jxnl]

right but what about instructor would make it easier, would love to see some pseudo-code

[Phodaie]

The current functionalities of Instructor are supper useful. If the same functionality (structure data extraction etc.) is ported to Assistant API, it can be used for extracting structured data etc. in apps that are already based on Assistant API.
It can wrap client.beta.threads.run.create() function.

[PaulGilmartin]

@jxnl For me, the applications of instructor to the assistants API would be identical to that of the chat completions API. I have a chatbot application which uses a graph of assistants, which essentially function as a decision tree: we have a collection of assistants used to attempt to classify what a prompt is about and then a final assistant which answers it. For the classification assistants, it's important that their answer is one of a predefined set of answers we supply (essentially a multiple choice). Right now to achieve this we need to add information to the prompt to ask the LLM to do this, something like:

"Choose one of the options below. Include no other text in your answer".

This proves to be unreliable, especially for models below gpt4 - often the LLM adds extra detail, breaking the expected output structure.

For this type of classification problem, which is I imagine a common use case for the new assistants API, having a library which enforces structured output would be amazing!

[Phodaie]

I think chat api will be deprecated in the next few months as the name implies it was initially intended for a narrow use cases.
Instructor is an elegant minimal framework as compared to Marvin, LangChain ... and is suitable for developers who want to build the rest of the functionality themselves. Many of the use cases are not obvious now but will come up over time.

Simple implementation could be at Assistant creation/update point and at run creation time by adding it as tool and instruction.

[jxnl]

😭 this is so much responsibility. Open to some sketch prs!

[covitof]

I have played around with assistants and have modified instructor locally to have some things (e.g. something similar to from_response) work with the assistants framework. I'd be happy to discuss/contribute if needed!

[jxnl]

would love to see that!

[PaulGilmartin]

@covitof I'd be interested in seeing that if you're able to show us

[covitof]

Basically I've used the assistants as described in the documentation and then, once I get the RunResult object from an assistant run (sort of equivalent to the response from a chat completion) I make use of the method from_run I've added to the OpenAISchema. This is basically a copy of from_repsonse with minimal changes. Here's the code

    @classmethod
    def from_run(
        cls,
        run: Run,
        tool_index: int = 0,
        validation_context=None,
        strict: bool = None,
    ):
        """Execute the function from the result of an assistant run

        Parameters:
            run (Run): The result object of an assistant run
            tool_index (int): The index of the tool call to execute
            validation_context (dict): The validation context to use for validating the response
            strict (bool): Whether to use strict json parsing

        Returns:
            cls (OpenAISchema): An instance of the class
        """
        submit_tool_outputs = run.required_action.submit_tool_outputs

        tool_call = submit_tool_outputs.tool_calls[tool_index]
        assert (
            tool_call.function.name == cls.openai_schema["name"]
        ), "Tool name does not match"
        return cls.model_validate_json(
            tool_call.function.arguments,
            context=validation_context,
            strict=strict,
        )

The type Run is from openai.types.beta.threads.run. I've stripped the method of all the checks just because I wanted to see whether this worked, but they could be added later. This basically works exactly like from_response would.

[covitof]

Something I think would be great is to have some functions that make the use of assistants as simple as using the chat completion. Though great, these assistants are a bit of a pain to manage, as all the steps needed to communicate with it are separate: add a message to the thread, let the assistant reply, poll the API to see when the answer is ready, fetch the answer, add the tool output to the thread, etc.. These though could be automatized by simple functions (I have a basic version of this as well). But I'm not sure if it makes sense to add this to instructor...

[PaulGilmartin]

@covitof Thanks for sharing the code. I actually haven't used instructor yet (because it doesn't support the Assistants API) so I'm not as familiar with the internals - would your patch above allow us to define a custom pydantic model as the output structure of a run result?

I also agree the current assistants API is a little convoluted. I think however this is likely something OpenAI themselves will simplify over time and agree it isn't really the job of instructor to do that.

[covitof]

Yes, exactly like instructor is intended to work for the normal chat completion. As long as your pydantic model inherits from instructors' OpenAISchema, you'd be able to do

run = client.beta.threads.runs.retrieve(run_id=run.id, thread_id=thread.id)
result = custom_pydantic_model.from_run(run)

And result would be the actual pydantic model constructed from the user's prompt, given that everything else is set up correctly. You can then handle different custom pydantic models etc, but obviously outside of isntructor.

[jxnl]

where does the response_model go eventually? at the agent model?

[covitof]

I have not yet looked into the patching itself as in my framework I deal with the problem of possibly invalid schemas in a different way. During the assistant creation (what I believe you refer to as agent model) only the available tools, system message and other very general properties are provided, so the response_model should be checked elsewhere.
Below some generic code to show the different steps (to be on the same page) and continue the discussion

from instructor import OpenAISchema, patch, Mode
from openai import OpenAI
client = patch(OpenAI(), Mode.TOOLS)


class UserDetail(OpenAISchema):
    age: int
    name: str

# create assistant
assistant = client.beta.assistants.create(
  name="User Detail Assistant",
  description="You are great at extracting name and age of users.",
  model="gpt-3.5-turbo-1106",
  tools=[{"type": "function", "function":UserDetail.openai_schema}],
)

# create thread
thread = client.beta.threads.create()

# add a message to the thread
client.beta.threads.messages.create(
    thread_id=thread.id,
    content="This person is called James and is 28 years old.",
    role="user",
)

# run the completion
run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id
)

# wait a bit of time...

# retrieve the completed run object
run = client.beta.threads.runs.retrieve(run_id=run.id, thread_id=thread.id)

print(UserDetail.from_run(run))
"""Output: UserDetail(age=28, name='James')"""

As you see from the code, the place where a response is actually created is only at the run step. The main technical problem, though, is that client.beta.threads.runs.create only posts the run request to the API: the run object has to be retrieved once the run has been completed (a simple polling loop is enough) to actually get the results. This means that the wrapping for allowing the response model can't really go in the run creation. Maybe a polling loop with the response_model checking onto the output could work? I'm not sure as, again, I don't use the response_model as I solve the problem differently.

Edit: I've just looked into the code for the chat completion wrapping and it's exactly what I was doing independently :D In the assistants API there is the function client.beta.threads.runs.submit_tool_outputs which is used explicitly to upload tool outputs, which could be used to feed back in the errors in case the assistant's output is not a valid schema. I'm not entirely sure, but one could actually wrap the run.create in such a way that it behaves like a chat completion call and returns the "final" result. It's probably a bit tedious though, not sure if you want to add it to instructor :)

[Phodaie]

I wonder just using create_and_run would work.

from instructor import OpenAISchema, patch, Mode
from openai import OpenAI
client = patch(OpenAI(), Mode.TOOLS)


class UserDetail(OpenAISchema):
    age: int
    name: str

# create assistant
assistant = client.beta.assistants.create(
  name="User Detail Assistant",
  description="You are great at extracting name and age of users.",
  model="gpt-3.5-turbo-1106",
  
)

run = client.beta.threads.create_and_run( 
    assistant_id=assistant.id,
    instructions = "This person is called James and is 28 years old.",
    tools=[{"type": "function", "function":UserDetail.openai_schema}],
)

sec = 0
while run.status == "queued" or run.status == "in_progress":
        run = client.beta.threads.runs.retrieve(
            thread_id=run.thread_id,
            run_id=run.id,
        )
        sec += 1
        print(f"Waiting for run to complete ({sec}s)" , run.status)
        time.sleep(1)

print(UserDetail.from_run(run))

client.beta.threads.delete(thread_id=run.thread_id)
#cancel since no need to wait for run to complete
client.beta.threads.runs.cancel(run_id=run.id)
client.beta.assistants.delete(assistant_id=assistant.id)

Possible you can wrap the above logic in a single function: extract(content : str , schema : OpenAISchema) . Also async version since you have to wait for the run status to transition to "requires_action".

[andrea-mucci]

Any Update about Assiatant APi Integration?