Support local LLMs such as CodeLLaMa (llama-2 based)

[aaronkaplan]

Hi!

I contributed a tiny set of patches so that gepetto can also access locally running LLMs such as CodeLLama (or see here).

• Added documentation to the README.md file
• Added sample config in config.ini
• Added code to access it.

Ping me pls. in case you want to test it, I can demo it if needed.

Have a good week!

[JusticeRage]

Hi! Thanks a lot for this PR! Let me try to run codellama-2 locally if it's not too complicated, to see if I can test the code and make it work!

[aaronkaplan]

You are welcome! It takes a bit of resources (I am running it on an A100 GPU!) . You could try with oobabooga in llama.cpp mode (read: on CPUs / mixed GPU+CPU) in case you don't have the resources.
Or shoot me an email (see the link in the README.md) and we can set up a call so that I can demo it to you.

Note that sometimes codellama does not return proper JSON yet. That's something which probably could be addressed with things like langchain's output parsers.

[JusticeRage]

Hi! So, I've started playing with the code, to check if the UI works, etc. Seems to be good, but just trying to send a request with the default config value (http://3.78.xxx/v1), I was surprised to get a reply from... OpenAI! Is that normal?
Also, I see a few # XXX and FIXME, is this version ready to be merged or is it still a work in progress?

[aaronkaplan]

Hi! So, I've started playing with the code, to check if the UI works, etc. Seems to be good, but just trying to send a request with the default config value (http://3.78.xxx/v1), I was surprised to get a reply from... OpenAI! Is that normal? Also, I see a few # XXX and FIXME, is this version ready to be merged or is it still a work in progress?

Let me check... it might be that you discovered a developer hack which was still in there.

[ghost]

how to adapt to https://huggingface.co/Phind/Phind-CodeLlama-34B-v2

[ghost]

Tested. It works! Thanks!