How to use with node-llama-cpp for backend

[TimeLordRaps]

Documentation description
Just a relevant example of how to use with node-llama-cpp
Steps include installing node-llama-cpp
installing langchain/community
and here's relevant code:

import { CopilotBackend, LangChainAdapter } from "@copilotkit/backend";
import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp"

const modelPath = ""
const contextLength = 8192
export async function POST(req: Request): Promise<Response> {
  const copilotKit = new CopilotBackend();
  return await copilotKit.response(
    req, 
    new LangChainAdapter(async (forwardedProps) => {
      const model = new ChatLlamaCpp({
        modelPath: modelPath,
        contextSize: contextLength,
        batchSize: contextLength, //weird bug with node-llama-cpp where if batchSize isn't big enough it just crashes.
      });
      return model.stream(
        forwardedProps.messages,
        { tools: forwardedProps.tools }
      );
    })
  );
}

Relevant Context
Where this documentation fits in

1. Existing documentation (see docs.copilotkit.ai): 'either https://docs.copilotkit.ai/reference/CopilotBackend) or https://docs.copilotkit.ai/getting-started/quickstart-backend

[mme]

@TimeLordRaps that's awesome, thanks for figuring this out!

Would you mind contributing this to our documentation?

[TimeLordRaps]

So I'm still currently testing around with this, it might not be fully functional as the LlamaCpp custom Llm form langchain has trouble with tools. Could someone familiar with the matter explain to me where in the code to look to see the downstream effects of failing tool uses.

[ataibarkai]

@TimeLordRaps at a high level the code lives here and here.

But it can be a bit challenging to come in there cold. It's a complex part of the codebase (and will actually get more complex in the near as we repurpose the underlying function calling capability to do more and more..).

It might be easiest to test empirically, having 2 identical examples and switching only node-llama-cpp and observing the differences.

I actually think this should live more than just in documentation, but in ready-made adapter factories / instances.

I'd love to collaborate / pair program with you to help you make this first addition of its kind of you are interested. And I can also help you get oriented with tool calls if you want to dig deeper.

btw, cool username

[TimeLordRaps]

I'd love the opportunity to pair program an implementation of some contribution to this project. However I'm not sure this issue would be my first choice. I'll reach out on discord though.

[mme]

This would make a great first issue: Add the code to the langchain docs to explain how to use other LLMs.