Documentation of LLM and Embedding Configurations

[donbr]

Thought I should share the documentation below with the team - I learned a lot going through the clear and concise architecture models that made it easier to get the configuration working.

I documented all LLM and embedding configuration in the same config.toml file as I wanted to do comparisons and standardization between OpenAI and the other models.

• The main issue I ran into was with the Anthropic model and I suspect that is a solvable embedding issue.
• I had similar issues with Ollama until I changed the embedding model.
• Groq / mixtral will likely benefit from a different embedding as well.

The team has built an incredible agentic system. I'm most impressed with your . In an open source solution, making sure people have a common vision of what is being built is priceless.

# config.toml file
# to get better results from certain LLM models may require additional embedding options.  Anthropic's recommendation to use Voyage AI is an example.
# full list of embeddings supported by LiteLLM is here:  https://docs.litellm.ai/docs/embedding/supported_embedding
# LiteLLM supported embeddings as of 3/31 are: OpenAI, Azure OpenAI, Bedrock, Cohere, Huggingface, Mistral AI, Vertex AI, Voyage AI

WORKSPACE_DIR="./workspace"

# ollama - https://docs.litellm.ai/docs/providers/ollama
LLM_API_KEY="sk-111111111111111111111111111111111111111111111111"
LLM_MODEL ='ollama/dolphin2.2-mistral:7b-q6_K'
LLM_EMBEDDING_MODEL="llama2" # can be "llama2", "openai", "azureopenai", or "local"
LLM_BASE_URL="http://localhost:11434"

# openai - https://docs.litellm.ai/docs/providers/openai
# LLM_API_KEY = "sk-111111111111111111111111111111111111111111111111"
# LLM_MODEL = "gpt-4-0125-preview" # cheaper alternative than the default ChatGPT4 model.  appropriate in certain circumstances.

# anthropic - https://docs.litellm.ai/docs/providers/anthropic
# AnthropicException - Number of request tokens has exceeded your rate limit (about 50% of the time)
# LLM_API_KEY="sk-ant-api03-xxxxx-xxxxx_xxxxx-EA-xxxxxx"
# LLM_MODEL="claude-3-haiku-20240307"
# LLM_MODEL="claude-3-sonnet-20240229"
# LLM_MODEL="claude-3-opus-20240229"

# groq - https://docs.litellm.ai/docs/providers/groq
# LLM_API_KEY="gsk_xxxxx"
# LLM_MODEL="groq/mixtral-8x7b-32768"

# plant uml docs covering state management increase the value of this "open source" project significantly
# the README is great - will make perfect sense once developers start looking closer at the tradeoffs between LLMs for their use case.
# referencing the full list of LiteLLM providers to map LLM models properly is essential:  https://docs.litellm.ai/docs/providers
# Claude 3 Opus provided some good examples of python coding challenges using this prompt:  "provide examples of industry standard tests used to validate generative ai system capabilities to generate python code.  For each standard, provide sample prompts that can be used ot evaluate the AI system."

[mklebrasseur]

@donbr I'm not sure if you noticed but theres API keys in your comments. If they're active keys you may want to rotate them now! If they're not active ignore me just looking out for fellow contributors!!

[donbr]

Thanks @mklebrasseur. This is huge... appreciate you catching my cut and paste failure and saving me big $$$. I guess this is what happens when you stay up late geeking out about a very cool project.

[gorbabor]

Thanks

[feirorum]

Maybe could be usable to have LM:s check our submissions online and warn for stuff looking like API-keys, credit card numbers, phone nr etc... run locally as a browser plugin maybe? I guess security folks are looking into how to enforce this for employees - and probably someone starts building fake ones w purpose to collect the info... :-)

[donbr]

Anthropic caught it and notified me.

[aSocialMenace]

Having an issues getting it to connect to my ollama server. is this correct? "WORKSPACE_DIR="./workspace"

ollama - https://docs.litellm.ai/docs/providers/ollama

LLM_API_KEY="sk-111111111111111111111111111111111111111111111111"
LLM_MODEL ='ollama/codellama'
LLM_EMBEDDING_MODEL="llama2" # can be "llama2", "openai", "azureopenai", or ">
LLM_BASE_URL="http://localhost:11434""

been fighting issues just getting it up and running.. npm gave me for some reason as im running linux on a windows sub system.. Any help would be awesome!

right now server is up and running but a big read warning window at the top says "ERROR: Failed connection to server. Please ensure the server is reachable at ws://localhost:3000/ws."

Would OObabooga be an option? could i run a llm in OObabooga instead of using ollama?

[dchambers]

My understanding is that WORKSPACE_DIR is what's exposed to the Docker sandbox, so if you want OpenDevin to update some repos on your machine you'd use WORKSPACE_DIR to point it to the directory containing those repos.

[dchambers]

Oh, regarding the big red box, I had that too and in my case it was because it couldn't interact with the Docker container sandbox as my Docker needed sudo to interact with it, and fixing that made it go away. It's a very generic error message though, so I guess it could be other things too.