Make a simple agent with Guidance and local LLMs

The Guidance is a tool for controlling LLM. It provides a good concept to build prompt templates. This repository shows you how to make a agent with Guidance. You can combine it with various LLMs in Huggingface. My medium article for more explanation.

UPDATE: Added gradio UI.

Install

Python packages:

• guidance
• GPTQ-for-LLaMa
• langchain
• gradio (Only for web UI)

Note: we only use langchain for build the GoogleSerper tool. The agent itself is built only by Guidance. Feel free to change/add/modify the tools with your goal. The GPTQ-for-LLaMa I used is the oobabooga's fork. You can install it with this command.

Run

There are two options: run a Gradio server with UI and run the notebook file.

Gradio server

Please modify the SERPER_API_KEY, MODEL_PATH, CHECKPOINT_PATH in the app.py and run:

gradio app.py

Notebook

Please check the notebook file. You should have a free SERPER API KEY and a LLM model to run this. I use the wizard-mega-13B-GPTQ model. Feel free to try others.

Example