Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What files are compatible? #124

Open
bedcoding opened this issue Mar 23, 2024 · 0 comments
Open

What files are compatible? #124

bedcoding opened this issue Mar 23, 2024 · 0 comments

Comments

@bedcoding
Copy link

I have read the document below.
https://llama-node.vercel.app/docs/start

So, I looked for a model to use in the source code at the site below.
https://huggingface.co/models?search=ggml

Then an error like this appears. What files should I get?

llama.cpp: loading model from model/llama-2-7b-chat.ggmlv3.q2_K.bin
error loading model: unrecognized tensor type 10

llama_init_from_file: failed to load model
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: Failed to initialize LLama context from file: model/llama-2-7b-chat.ggmlv3.q2_K.bin] {
  code: 'GenericFailure'
}

code:

import { LLM } from "llama-node";
import { LLamaCpp } from "llama-node/dist/llm/llama-cpp.js";

const model = "model/llama-2-7b-chat.ggmlv3.q2_K.bin";
const llama = new LLM(LLamaCpp);
const config = {
    modelPath: model,
    enableLogging: true,
    nCtx: 1024,
    seed: 0,
    f16Kv: false,
    logitsAll: false,
    vocabOnly: false,
    useMlock: false,
    embedding: false,
    useMmap: true,
    nGpuLayers: 0
};

const template = `How are you?`;
const prompt = `A chat between a user and an assistant.
USER: ${template}
ASSISTANT:`;

const run = async () => {
  await llama.load(config);

  await llama.createCompletion({
      nThreads: 4,
      nTokPredict: 2048,
      topK: 40,
      topP: 0.1,
      temp: 0.2,
      repeatPenalty: 1,
      prompt,
  }, (response) => {
      process.stdout.write(response.token);
  });
}

run();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant