Could you please showcase one end-to-end example to run a model over Kubernetes cluster (AKS, etc) with one RAG implementation please?

[ajmal-yazdani]

I can download the model locally and can write a simple chat application.

But question is how we can run this model over Kubernetes cluster and run the RAG application.

Could you please guide with some sample?

[doberst]

Great feedback - and definitely something on our near-term roadmap. Will come back to you soon. Also, let us know if you would be interested potentially to collaborate with us on building up the Kubernetes script around this use case.

[chair300]

@ajmal-yazdani This is a great suggestion. I have been working on a websocket chat agent already. While its not ready for checkin and/or demo yet it will be coming soon.

The secondary point about sample kubernetes code to deploy and run RAG is very interesting. Some of the kubernetes code is dependent of the specific use case and model since the model is embedded within the application. I am happy to put together very simple kubernetes deployment code bases on docker containers (I need to build the containers first).

I hope to have the kubernetes code by mid January, with the docker containers built and available before then.

[ajmal-yazdani]

Great feedback - and definitely something on our near-term roadmap. Will come back to you soon. Also, let us know if you would be interested potentially to collaborate with us on building up the Kubernetes script around this use case.

Something for sure I can try on my AKS cluster or local docker.

[ajmal-yazdani]

Thank you very much!

[chair300]

I just updated the docker image which can run llmware examples in a aks cluster or local cluster. There is a docker-compose file which provides the extra database infrastructure. You will still need to setup the env's but its a start.

[chair300]

I have a pull request outstanding which will allow you to play with a full docker-compose full service infrastructure. Please see my pull request for the update.

[chair300]

with docker-compose from the devcontainer folder you can run the following command and then all the infrastructure will be up. Docker exec -it into the llmware container and run the examples. Its all set and easy. Please let me know if you have any further questions.