Is llm still necessary, and should development still continue?

[philpax]

Development of llm hasn’t gone as quickly as I would have liked. I have been very busy with work over the last few months, and keeping up with the pace of development in the space has been quite difficult.

There is no better demonstration of this than the GGUF implementation work - after writing up the spec and getting everyone to agree on it, I haven’t had much energy to work on actually implementing it (#412), especially in accounting for how the tokenizer has changed and handling the other model architectures. I’ve been fixing things up here and there, but it’s still a while off.

All of that would be fine - we could always catch up - if it weren’t for the existence of Candle, or more specifically, candle-transformers. Like its Python namesake, candle-transformers contains implementations of common models, including the majority of, if not all of, the models we support.

My plan was to always support Candle as an additional, or replacement, backend. However, I did not account for Candle itself implementing these models and including both support for GGUF/GGML models and quantisation in such a short timeframe, allowing it to more or less cover the same territory as we do. Additionally, it enjoys a dedicated full-time developer who can dedicate their time to progressing Candle much more expediently than any one of us can progress llm.

Given that, I’m wondering if we still need to be around. I’m not convinced that a library that is neither bleeding edge (llama.cpp) or ecosystem-native (Candle) can carve out its own unique place in the Rust ML ecosystem.

I’ve heard some suggestions that we could offer a more ergonomic interface on top of CT, but I suspect that effort would be better spent on making CT more ergonomic in itself.

I’ve created a poll with a few options, but I’d love to hear any detailed thoughts people have, reports from production use of the library, or reasons why llm should stick around.

In any case, I appreciate all of our users - I love seeing what people have done with the library! - and hope that we have been of use to you. Thanks for sticking around!

[iashchak]

First and foremost, I extend my heartfelt gratitude for the outstanding contribution to the llm project.
You've set an unparalleled standard for the experience with non-llama ggml models currently available.
I was profoundly impressed by the immediate, out-of-the-box operability.
Is CT indeed feature-rich at this moment?
Irrespective of your forthcoming decisions, I am thankful for the provision of such an impeccable tool.

[philpax]

Thank you for the feedback! I really do appreciate it 🥰

You can have a look at what CT supports in the main repo's README: https://github.com/huggingface/candle/tree/main#check-out-our-examples. There are a few nuances there (they treat quantized models as a separate class of models from non-quantized models, from what I can see), but you can see that it more or less covers what we do (unfortunately, I don't think there are many users of BLOOM these days).

[iashchak]

And they don't support gpt-2:)

[philpax]

No, it doesn't look like they do. My guess for their motivation not to implement it is that there aren't that many uses of GPT-2 that couldn't be replaced by a more recent architecture - other than, of course, supporting existing GPT-2-based models.

[pabl-o-ce]

depends of your goals you already inspired many people by example!

[steventrouble]

Wow, that's wild. When I first started, llm was far and away the best Rust library for CPU inference.

But I haven't been in the space for a few months. If another library's come out with all the same features, I wouldn't be too mad at having to use the new one.

I think it's better when we all contribute to the same project rather than splitting our efforts. So if you're already slowing down, I think it's fair to recommend the other. YMMV.

I'm just grateful that this project was around when I needed it. Thanks!

[WeirdConstructor]

I think you should decide how you want to spend your spare time.

But I want to leave a big thanks ♥️! You picked up the project from it's prototype state and built a nice library/architecture around it. Back then there was no candle or candle-transformers.
You also started the GGUF effort in the first place, the effort writing that up and discussing it is an amazing contribution to the whole ecosystem.

If you feel that candle is going to the right direction and pushes much faster ahead, there is no shame in directing people towards that project. Especially if they got a full time dev behind it and you don't feel like trying to catch up.

[danforbes]

I voted Other as I believe we must consider feedback from the Candle implementers before making our decision.

[philpax]

Done: huggingface/candle#1186

[nerdypepper]

for me, the biggest win with using llm over other implementors has been the ggml backend, which indirectly offers gpu inference on cuda and metal. so i suppose my opinion lies in between a couple of options posted above:

• i see great value in rust bindings to ggml. admittedly, this is tough to maintain give the velocity of llama.cpp (and therefore ggml)
• candle might be the way to go to write inference code for newer models, give how tedious it is to develop on top of ggml

[benbot]

I think independent (non corporate backed) libraries are very important and worth working on, even at a slower pace.

Competition is good in open source too.

[Aisuko]

Hi, thanks for the amazing project. I have some experience with the open-source AI project. And I'd like to share some thoughts here. Here are more ML frameworks in Rust, like you mentioned candle, burn-rs, etc. They can support many of the LLMs. And these projects have their own goal. Please think about it, why do we need a new framework since we already have PyTorch? And the answers are quite different due to the different projects.

So, here is my thought. What is the roadmap or goal of the rusinfomer/llm? If you want to build an open-source AI ecosystem with Rust. I believe it should be continued. And support other ML frameworks step by step. And I believe there are gaps between those frameworks and the end-user. And projects like rusinfomer/llm can help with this. Although this is a long story, being more patient is okay. For example, I am trying to embed burn as a backend to some project that only has 0.1% in the past 2-3 weeks.

I believe we are fighting for the open-AI technology not for maintaining the API or the CLI (inspired by other maintainers, not mine). So, as I mentioned before. You can stop a little bit to think more about the goal of the project.

By the way. I believe I can make some contributions after I am familiar with Rust ML frameworks. So, keep fighting.

[philpax]

Thanks to everyone for the insight, feedback, and motivation. After reading through what everyone's said, talking to a few users, and chatting with Laurent regarding Candle and candle-transformers, I'm happy to state that the project will go on 🚀

To list the reasons I came to this decision:

• Development of llm is independent. I'm not too concerned by this (Candle has the same license and HF have been good stewards), but it's worth noting.
• llm's API is higher-level than candle-transformers, which reduces the barrier to entry to playing with LLMs.
• llm has a unified API that makes it easy to try out other models, or use the same set of operations across multiple models that you may be using simultaneously.
• Our use of GGML allows for Metal support and higher performance than Candle under some circumstances.
• Multiple attempts at the same problem allows for more exploration of the design space, so that the ecosystem can explore different options and compromises.
• The roadblocks are not insurmountable. We'll be able to catch up to the current state of affairs of open-source LLMs with some work.

That being said, I'm looking forward to a harmonious relationship with Candle - I would love to implement Candle as an alternate backend (#31), as it'll enable easy use of LLMs on platforms where GGML may not be ideal or available, as well as simplifying the build. We'll see what the future has in store for us 🙂

Thanks again to everyone weighing in - your support was crucial to this decision!