web-llm integration

[louis030195]

Feature Description

Compatibility with

https://github.com/mlc-ai/web-llm

Use Case

Running LLM in the browser, no need a server

Additional context

I've been using ai with mlc-llm (core lib powering web-llm) by running as a server by implementing a provider (very similar to the Mistral one) but I want to just run the LLM in the browser so there is less infra, and it can still leverage client's GPU

Anyone tried to use ai with web-llm?

[lgrammel]

You could implement a custom provider (see https://sdk.vercel.ai/providers/community-providers/custom-providers ) that directly calls web llm. The llamacpp community provider does something similar (in process communication) I think.

[louis030195]

@lgrammel i thought about this but isn't provider called on server side?

basically wants to leverage the hooks like useChat, generative UI stuff etc. but with client side LLM

[lgrammel]

AI SDK core could also be used on the client side. That said, you are right, useChat etc require a server connection. What you could do is use e.g. AI SDK Core streamObject or streamText client side and then operate directly on those results (without useChat et al).

[louis030195]

Is this sdk going to have better support for client side LLMs? Clients have more and more powerful AI accelerators, future apps will use both client and server side LLMs

Got it working but too much effort to make it work with all the generative UIs features atm which assume a lot of server side stuff.
Depending update of this issue will implement my own lib for client side generative UI