Skipping USER when using a tool

[mdelmans]

Is your feature request related to a problem? Please describe.
When I use tools, the sequence of messages goes:

• USER asking a question
• LLM responding with a TOOL call
• AGENT responds with a TOOL result
• USER is prompted for a response

Describe the solution you'd like
I think the right (at least for my use) sequence should be:

• USER asking a question
• LLM responding with a TOOL call
• AGENT responds with a TOOL result
• LLM responds to the original question using results from the tool

So USER should not be added to the list of responders in Task.step() if pending message is a response from the AGENT.

Describe alternatives you've considered
I know I can skip human prompt. Is there another way? Maybe there is a way to tag a tool such that it adds "LLM" as a recipient?

Additional context

[pchalasani]

Maybe there is a way to tag a tool such that it adds "LLM" as a recipient?

Indeed, there's a SEND_TO routing you can add as prefix to the string returned by the tool handler method, see example here:
https://github.com/langroid/langroid/blob/main/examples/docqa/extract-then-chat.py#L76

We haven't gotten around to documenting this yet.

[pchalasani]

Incidentally, we don't want to enforce that after agent_response() the result should always be handled by llm_response() -- one reason is that the result of a tool could be a huge amount of text (e.g. think of a sql query result) and we want to allow user to intervene (when interactive=True) to either allow the LLM to respond, or insert their own message

[mdelmans]

@pchalasani thanks for the response! Will give it a go.

I'm new to the field, maybe we are thinking about different use cases. I am interested in RAG, where I don't want user to see/act upon the response from a tool/function. As far as user is concerned, tool is just a part of the agent not a separate entity. I want LLM to transform the response and tailor it to the original user prompt. Huge amount of text, in this case, should be handled by the tool/function itself. For example, limit the result of the sql query to a manageable size.

Don't get me wrong, I think SEND_TO is a perfect solution. Maybe worth wrapping it (optionally) so it is added automatically to the tool response.

[pchalasani]

Huge amount of text, in this case, should be handled by the tool/function itself.

I only listed on possible reason (i.e. result size) why we'd want the user to have a chance to "intervene" before an LLM sees a tool result. As you said this scenario can be handled by putting the burden on the tool handler method to cut the result size (which still may not be perfect, since the handler method would potentially need to do some parsing etc).

Another reason for user intervention could be that the tool result may contain sensitive info, and the user gets a chance to check and override.

So we made a design choice here: instead of creating a special rule ("send all tool results directly to LLM") and have possible undesirable scenarios where results shouldn't be seen by the LLM, we rather put the onus on the developer who is writing the tool handler, to add a SEND_TO to explicitly send it to LLM if they are confident that is fine to do, in their specific scenario.

Another way is to have two separate agents. Assistant directly interfaces with the user, takes their high-level questions, and generates questions to RagAgent (which would have interactive=False). See an example of this in examples/docqa/doc-chat-2.py and other similar variants.

[pchalasani]

Maybe worth wrapping it (optionally) so it is added automatically to the tool response.

This could be one way, e.g. via a config flag. We're thinking about this or other good ways to let the developer control message flow in various states.

[pchalasani]

Forgot to mention one other thing. When you set up a task with interactative=True, then it will wait for user input after each valid response from a non-human entity (i.e. LLM or Agent tool handler). One big reason we do it this way is to avoid infinite loops between LLM and AGENT responses -- e.g.

• LLM generates a tool,
• tool handler returns result or err
• LLM generates tool again
• tool handler returns result or err
• ...

If you really want a scenario where it takes your first question, and then subsequently becomes non-interactive, then you can either use SEND_TO or a multi-agent set up as mentioned above.
Another possibility is you could write your own loop:

task = lr.Task(..., interactive=False)
while True:
    query = ... prompt user for their question...
    result = task.run(query)