Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function calling #996

Merged
merged 93 commits into from
May 23, 2024
Merged

Function calling #996

merged 93 commits into from
May 23, 2024

Conversation

nsarrazin
Copy link
Collaborator

@nsarrazin nsarrazin commented Apr 9, 2024

**Screenshots**
  • Probably only works well with command R+, requires a tokenizer with chat prompt templates that has multiple prompt templates for tools
  • Hardcoded a few tools
    • websearch
    • text 2 img
    • calculator
    • url fetcher (injects url content as plaintext)

@nsarrazin nsarrazin added enhancement New feature or request front This issue is related to the front-end of the app. back This issue is related to the Svelte backend or the DB assistants Related to the assistants feature labels Apr 9, 2024
@flozi00
Copy link
Contributor

flozi00 commented Apr 10, 2024

The pr looks very promising, I only have some open questions about integration.
At the moment there are built in tools with static specified functions behind, what's the workflow to be expected for adding more tools ?
Can will there be a list of links, local mounted folder with codes or a moderated pool of tools within the main repo ?

I already have some tools to be added and would love to contribute them

@wdhorton
Copy link

It would be cool if users could specify a tool by pasting in an OpenAPI spec

@wdhorton
Copy link

Really excited for this feature

@nsarrazin
Copy link
Collaborator Author

Hi! So we're thinking about how to best integrate this feature. We'll probably add a few built-in tools that call some nice models from the hub as a first step, and then add support for OpenAPI tools afterwards since that's quite a bit more challenging to do, but it's definitely on my roadmap.

If you have any suggestions, tips or use cases for this feature, please let me know, it's always useful to know how people plan to use a feature 😄

@nsarrazin
Copy link
Collaborator Author

I added a text2img tool, seems to work well. Now i'm adding:

  • A component to open an image when you click on it (right now it only shows a thumbnail)
  • A component for displaying & playing audio files

That way we'll have good multimodal support for tools 😄

@gururise
Copy link
Contributor

Would love to see a tool that can perform RAG given a url to a chromaDB.

Saghen and others added 14 commits May 21, 2024 21:59
* details: close on click outside

* drag for tools

* tools ordering

* add link to community discussion

* misc

* calculator listed as last

* file style

* images with different ratios

* page scraper: longer timeout

@Saghen @mishig25 seems to work better with this

* vertical gap

* Revert "drag for tools"

This reverts commit 8eeed3b.

* chat padding on desktop

* file colors

* larger gap for all messages on xl screens
{#if searchUpdates && searchUpdates.length > 0}
<OpenWebSearchResults
classNames={tokens.length ? "mb-3.5" : ""}
webSearchMessages={searchUpdates}
/>
{/if}

{#if toolUpdates}
{#each Object.values(toolUpdates) as tool}
{#if tool.length > 0}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{#if tool.length > 0}
{#if tool.length}

Copy link
Collaborator

@mishig25 mishig25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after the CI is greaan, LGTM ! 🚀 🔥
Great job Liam & Nathan!

@mishig25 mishig25 changed the title Function calling for assistants Function calling May 23, 2024
@mishig25 mishig25 merged commit 989379c into main May 23, 2024
3 checks passed
@mishig25 mishig25 deleted the feat/functions branch May 23, 2024 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
assistants Related to the assistants feature back This issue is related to the Svelte backend or the DB enhancement New feature or request front This issue is related to the front-end of the app.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants