Embeddings model: bge-base-en-v1.5

[francis2tm]

Hello,
I was wondering if it's possible to run bge-base-en-v1.5 model with llama.cpp, and if yes, could anyone give me a breakdown on how to do it?

Thanks in advance!

[ianscrivener]

bge-base-en-v1.5 is not on the list of supported models.

Curious, what about this model makes it better than some of the many other models that llama.cpp already supports?

[francis2tm]

Indeed @ianscrivener, but this is an embeddings model. And a quite good at that: https://huggingface.co/spaces/mteb/leaderboard, currently the 2nd best embeddings model.
I wouldn't mind doing the support myself and later do a PR. But I would love a little guidance beforehand ;)

[ianscrivener]

Embeddings model. Right. I knew that last week but seem to have experienced a grey matter brown out 😳

May be you can tweak one of the examples as a POC?

Hopefully one of the C/C++ devs can chime in... this is above my pay grade.

[monatis]

Supporting embedding models is on the way, starting from BERT and its variants. And it will be accompanied by a RAG example directly in C/C++.

[francis2tm]

Awesome! Do you have a timeline for the RAG example?

[monatis]

Estimating one week or so.

[zerodrift]

Hi there. Any updates on this? sorry to bother you ... :)

[Johnhersh]

Kinda curious what happened with this?

[willie]

Oh man, what happened?

[xyzhang626]

hey @francis2tm @zerodrift and other guys, if you are still interested, I've implemented BGE series model with ggml. Check this embeddings.cpp.

[golangpkg]

can you merge embeddings.cpp into llama.cpp ?

[xyzhang626]

hey @golangpkg thanks for your interest. I believe there are some awesome guys have done this. Check this.

[phymbert]

The server has an embeddings endpoint openAI compatible. What do you need more ?

[wangzhaode]

hey @francis2tm
I've implemented bge-large-zh in project mnn-llm

bge-large-zh 8bit quant model speed can reach 749.12 tok/s on Apple M1 Soc with 4-thread.