Nano Jarvis is compatible with either a cloud-based (managed) LLM service (e.g. OpenAI GPT model, Grog, OpenRouter, etc) or with a locally hosted LLM server (e.g. llama.cpp, LocalAI, Ollama, etc). Please continue reading for detailed instructions.
Requirement: Node.js v18 or later.
Launch with:
./nano-jarvis.js
then open localhost:5000
with your favorite web browser.
Supported local LLM servers include llama.cpp, Nitro, Ollama, and LocalAI.
To utilize llama.cpp locally with its inference engine, ensure to load a quantized model such as Phi-3 Mini, LLama-3 8B, or OpenHermes 2.5. Adjust the environment variable LLM_API_BASE_URL
accordingly:
/path/to/llama.cpp/server -m Phi-3-mini-4k-instruct-q4.gguf
export LLM_API_BASE_URL=http://127.0.0.1:8080/v1
To utilize Nitro locally, refer to its Quickstart guide for loading a model like Phi-3 Mini, LLama-3 8B, or OpenHermes 2.5 and set the environment variable LLM_API_BASE_URL
:
export LLM_API_BASE_URL=http://localhost:3928/v1
To use Ollama locally, load a model and configure the environment variable LLM_API_BASE_URL
:
ollama pull phi3
export LLM_API_BASE_URL=http://127.0.0.1:11434/v1
export LLM_CHAT_MODEL='phi3'
For LocalAI, initiate its container and adjust the environment variable LLM_API_BASE_URL
:
docker run -ti -p 8080:8080 localai/localai tinyllama-chat
export LLM_API_BASE_URL=http://localhost:3928/v1
To use OpenAI GPT model, configure the environment variable OPENAI_API_KEY
with your API key:
export LLM_API_BASE_URL=https://api.openai.com/v1
export OPENAI_API_KEY="sk-yourownapikey"
To utilize other LLM services, populate the relevant environment variables as demonstrated in the following examples:
export LLM_API_BASE_URL=https://api.endpoints.anyscale.com/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Llama-3-8b-chat-hf"
export LLM_API_BASE_URL=https://api.deepinfra.com/v1/openai
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="mistralai/Mistral-7B-Instruct-v0.1"
export LLM_API_BASE_URL=https://api.fireworks.ai/inference/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="accounts/fireworks/models/llama-v3-8b-instruct"
export LLM_API_BASE_URL=https://api.groq.com/openai/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="gemma-7b-it"
export LLM_API_BASE_URL=https://mixtral-8x7b.lepton.run/api/v1/
export LLM_API_KEY="yourownapikey"
export LLM_API_BASE_URL=https://openrouter.ai/api/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="mistralai/mistral-7b-instruct:free"
export LLM_API_BASE_URL=https://api.together.xyz/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Llama-3-8b-chat-hf"