Function Calling Demo

[TheR1D]

Just wanted to share some progress on a feature I'm currently working function calling. I was surprised how many interesting things might be done using function calls.

sgpt_fn_calling_demo_lb.mp4

General idea is to implement some easy way to allow ShellGPT users to create custom functions for LLMs. I came up with the idea to create some kind of function "outline" Python class, which basically will describe a function for LLM with description, all parameters and the expecting output.

So here is some simplified workflow I came up with. ShellGPT has new "watch folder" which will store all functions .py files with some standardized Python classes. Before calling OpenAI, we are looking into folder, collecting the functions, and generating JSON schema to pass them to OpenAI when making request to the API. Simple example of a function class:

import subprocess
from pydantic import BaseModel, Field
from instructor import OpenAISchema


class Function(OpenAISchema):
    """
    Opens a web link (URL) in browser.
    """

    url: str = Field(..., example="https://example.com")  # Can be array.

    class Config:
        title = "open_url_in_browser"

    @classmethod
    def execute(cls, url: str):
        subprocess.run(["open", url])
        return "Success. URL is opened in browser."

Once we have it in our "watch folder" we can start using it right away, since ShellGPT will load them and pass to OpenAI API.

sgpt "provide last 5 links to PEP documentation and open them"

Output:

▌ @FunctionCall open_url_in_browser

▌ @FunctionCall open_url_in_browser

▌ @FunctionCall open_url_in_browser

▌ @FunctionCall open_url_in_browser

▌ @FunctionCall open_url_in_browser

Here are the links to the five most recent Python Enhancement Proposals (PEPs) at the time of my knowledge cutoff in April 2023:

 1 PEP 690 - Lazy Imports
 2 PEP 689 - Maintaining the Stable ABI
 3 PEP 688 - Streamlining the PEP process
 4 PEP 687 - PyPA Governance
 5 PEP 686 - Make os.urandom() blocking on Linux

Please note that new PEPs may have been published since my last update, and you can check the latest PEPs by visiting the PEP Index. All five PEP documentation links have been opened in your browser.

In general it open a whole new world of possibilities, for example if we have multiple shell commnds in the LLM output, it could execute them right away, and check the stdout/stderr and based on result take futher actions to complete the task.

Another example:

sgpt "play music"

Output:

▌ @FunctionCall play_music

The music is now playing. Enjoy the tunes! 🎶

(starts playing music in Apple Music according to the function class).

sgpt "what functions do you have?"

Output:

I have access to the following functions:

 • get_user: Retrieves a user record from a database based on the provided name.
 • send_telegram_message: Sends a message via Telegram.
 • play_music: Initiates music playback.
 • open_url_in_browser: Opens a specified URL in a web browser.
 • say_text: Converts text to speech and says it out loud.

I would greatly appreciate your feedback and any ideas you might have on this concept. Your input could be valuable in refining and enhancing this feature. Please feel free to share your thoughts.

[hcal]

That's amazing. I've been playing with ways to integrate a llms into my task management workflow with task warrior as the local back end storage. It has a robust Python Library So I wouldn't be surprised If that was easy to implement with function calling plugin. Another thing that might be nice is a generic notes storage plugin with something like a sqlite backend.

[TheR1D]

Some updates on function calls. Today I was working to make it possible for LLM to keep executing functions until it will accomplish given task. For example if we ask LLM to execute some shell commands, and they might fail, it can refine the command based on exit code, stdout/stderr.

sgpt --repl temp
Entering REPL mode, press Ctrl+C to exit.
>>> what is in this folder /Users/user/somewhere

▌ @FunctionCall execute_shell_command


 Arguments:
 {"shell_command":"ls /Users/user/somewhere"}
 Result:
 Exit code: 0

 Output:
 test.png
 test_1.png


The folder /Users/user/somewhere contains the following files:

 • test.png
 • test_1.png
>>> convert these images to jpg

▌ @FunctionCall execute_shell_command


 Arguments:
 {"shell_command":"sips -s format jpeg /Users/user/somewhere/*.png --out /Users/user/somewhere"}
 Result:
 Exit code: 0

 Output:

 Warning: /Users/user/somewhere/*.png not a valid file - skipping


It seems there was an issue with the command; the wildcard *.png was not processed correctly. I will attempt the conversion again, this time by converting each file individually.

▌ @FunctionCall execute_shell_command


 Arguments:
 {"shell_command":"sips -s format jpeg /Users/user/somewhere/test.png --out /Users/user/somewhere/test.jpg"}
 Result:
 Exit code: 0

 Output:
 /Users/user/somewhere/test.png
   /Users/user/somewhere/test.jpg


▌ @FunctionCall execute_shell_command


 Arguments:
 {"shell_command":"sips -s format jpeg /Users/user/somewhere/test_1.png --out /Users/user/somewhere/test_1.jpg"}
 Result:
 Exit code: 0

 Output:
 /Users/user/somewhere/test_1.png
   /Users/user/somewhere/test_1.jpg


The images test.png and test_1.png have been successfully converted to JPEG format with the filenames test.jpg and test_1.jpg, respectively, in the folder /Users/user/somewhere.

Explanation

First we asked LLM to get files in the directory, LLM handled it by simply calling ls.

ls /Users/user/somewhere

Now when we have list of files in the conversation history, I asked LLM to convert png files to jpg. GPT-4 first tried to execute:

sips -s format jpeg /Users/user/somewhere/*.png --out /Users/user/somewhere

It didn't convert png files and ended with warning:

Warning: /Users/user/somewhere/*.png not a valid file - skipping

Since we are passing output of the shell command back to GPT-4, it responded back with following message:

It seems there was an issue with the command; the wildcard *.png was not processed correctly. I will attempt the conversion again, this time by converting each file individually.

Folowed by two more shell commands.

sips -s format jpeg /Users/user/somewhere/test.png --out /Users/user/somewhere/test.jpg
sips -s format jpeg /Users/user/somewhere/test_1.png --out /Users/user/somewhere/test_1.jpg

This time both png files were successfully converted to jpg. The magic part of it that it was triggered by single LLM prompt:

convert these images to jpg

And it was executing shell commands one by one until it gets to the desired results.

[endolith]

Today I was working to make it possible for LLM to keep executing functions until it will accomplish given task.

So that means it can see the output of its own commands? That would be great

[schwab]

Can this be configured to work with ollama?

[velinxs]

wow amazing. Do you think this could be setup somehow as like an integration into n8n ? so that i could say type a command in slack then it would run on terminal in docker container somewhere?

[endolith]

This looks really powerful, but I don't want functions to be called immediately, I want to see them and confirm them first. Even in the example you gave, I might not want the AI to open up 300 tabs in my browser, for instance.

[velinxs]

I definitely want it to work on its own. Would be cool to like search Google with and feedback summaries, a scraper.

…

On Thu, Apr 4, 2024, 9:58 AM endolith ***@***.***> wrote: This looks really powerful, but I don't want functions to be called immediately, I want to see them and confirm them first. Even in the example you gave, I might not want the AI to open up 300 tabs in my browser, for instance. — Reply to this email directly, view it on GitHub <#416 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A66AEW6TDMVGCPTXB7KNEBLY3VTA3AVCNFSM6AAAAABBAWZ65SVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAMJQHE2TO> . You are receiving this because you commented.Message ID: ***@***.***>

[hcal]

I love this feature. GPT4 and 4o seem to be very aware of GNOME and KDE settings so you can say "open the settings for monitor scaling" and it pops open. You can also say "switch to gnome dark mode" and it just does the setting for you, which is pretty cool.

It might be nice, as an option, to open a non-interactive terminal so you can watch the command execute. That would let you know if its hung due to an error or still working.

[hcal]

Another potential improvement for functions would be a method to handle or avoid issues caused by interactive Bash programs that wait for user input. Prompts like "Are you sure?" will cause the function to hang because they don't return as finished while waiting for user interaction.

From the README:

The docstring comment inside the class will be passed to the OpenAI API as a description for the function, along with the title attribute and parameter descriptions. The execute function will be called if the LLM decides to use your function. In this case, we are allowing the LLM to execute any Shell commands in our system. Since we are returning the output of the command, the LLM will be able to analyze it and decide if it is a good fit for the prompt. Here is an example of how the function might be executed by the LLM:

Based on this, I tried added a note to the docstring informing the model that the shell commands are non-interactive and should be formulated accordingly. This seemed to help, and I didn't encounter the issue again. Later, just for additional clarity, I added an example list of common flags to the reminder, which also seems to work, though I haven't tested it as extensively.

"""
Executes a shell command and returns the output (result). Commands are non-interactive, so formulate shell commands to complete without user interaction where needed (e.g., -y, --yes, --assume-yes, --noconfirm, --batch, --no-prompt, --non-interactive, --accept-license, --auto).
"""