Put some actions in the notes with LLM

[ClemTB]

Hello guys,

I made a proof of concept few month ago for a LLM to make action (or define actions) based on the request of the user. My case was pretty simple:
I give the LLM a list of well known functions (like "start my music player", "stop the music", "create an alarm", ... with a name and a precise description) with arguments (name + precise description). Then I ask the LLM to respond with the best succession of functions that can fullfill the user's request with proper tags. Ultimately the functions were hyper links that you can click to execute the function directly.

I think Reor can use something like this to:

• Create new notes based on notes + LLM knowledge ("make me a new draft that summarise the data about my garden")
• Create new notes based on web search + LLM knowledge ("Create a note that teaches me about knots")
• Reorganise notes and folders ("Put some order in my 'To be treated' folder)
• Open a web browser to a pertinent web search based on notes ("Find some picture related to my notes 'Cats and trees'")
• Import files from external sources
• ...
  Sky is the limit.

From my perspective, "All you need to do" (a common dev joke) is to create a bunch of well known functions and tells the LLM to answer with proper tags if it feels like you asked for actions to be made. Then treat the output text to display the actions as hyperlinks or buttons that executes the related function with the right arguments.
In my opinion, if you ever implement this, let the user choose to execute the functions, I personally don't want the LLM to mess up with my data because it gets something wrong.

Hope this helps

[samlhuillier]

Sorry not sure how I missed this!

But absolutely. We will start by using the tool construct from OpenAI and provide basic tools like "Create note", "Edit note", "Rename note" etc. And I think the initial implementation will be to allow users to optionally toggle on these tools in chat.

[ClemTB]

Does that mean this feature will be available only when using openAI in Reor or will it work with local LLM too?

[samlhuillier]

No it's just that right now local models have json mode which is a slightly different construct to tool use. I definitely expect them to have tool use very soon...And if not, we can still do workarounds to make json mode be like tool use or vice versa :)