[Bug]: local models not able to be configured to work

[gbenaa]

Is there an existing issue for the same bug?

• I have checked the troubleshooting document at https://github.com/OpenDevin/OpenDevin/blob/main/docs/guides/Troubleshooting.md
• I have checked the existing issues.

Describe the bug

There are a number of issues open about whether local models are usable or not. It works fine making remote calls to Groq but I run out of credit really quickly, and I need to use my local gpus in any case.

I have tried for six hours today to get these working LOCALLY:
codellama:7b
llama3:latest

In Issue #1336 rbren says "In general, local models aren't powerful enough yet to run OpenDevin...." Well, that may be the case, but I have llama3:70b installed on my local machine, and four gpus to run it (ollama spreads the load nicely). I think that configuration will easily work for OpenDevin. But I cannot get it to work based on the instructions.

A am not really asking you to fix this at this point, but can you just confirm whether there is a known issue here?

If you're not aware of one, then I'm happy to carry on testing, and I'll report all the different configurations I've attempted. But I don't want to spend hours more testing if there is a known issue.

for example:

llama3
llama3:latest
ollama/llama3
ollama/llama3:latest

LLM_API_KEY=ollama
LLM_MODEL=ollama/llama3

Thanks for your help - I feel it's an amazing product - if only I could get it to work! :)

Current Version

I'm running in docker, the most recent version, using this command:
docker run \
	--add-host host.docker.internal=host-gateway \
	-e LLM_API_KEY="ollama" \
    -e LLM_BASE_URL="http://host.docker.internal:11434" \
	-e LLM_MODEL=ollama/llama3 \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    ghcr.io/opendevin/opendevin:main

Installation and Configuration

docker run \
    --add-host host.docker.internal=host-gateway \
    -e LLM_API_KEY="ollama" \
    -e LLM_BASE_URL="http://host.docker.internal:11434" \
    -e LLM_MODEL=ollama/llama3 \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    ghcr.io/opendevin/opendevin:main

Model and Agent

MonologueAgent
llama3:latest

Reproduction Steps

docker run
--add-host host.docker.internal=host-gateway
-e LLM_API_KEY="ollama"
-e LLM_BASE_URL="http://host.docker.internal:11434"
-e LLM_MODEL=ollama/llama3
-e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE
-v $WORKSPACE_BASE:/opt/workspace_base
-v /var/run/docker.sock:/var/run/docker.sock
-p 3000:3000
ghcr.io/opendevin/opendevin:main

(note the command above has also been run without
-e LLM_MODEL=ollama/llama3 )

browse to
http://localhost:3000/index.html

click gear on the RIGHT bottom side (not left)

type model name:
all these tried
llama3
llama3:latest
ollama/llama3
ollama/llama3:latest

Logs, Errors, Screenshots, and Additional Context

console output below:

==============
STEP 0

21:55:19 - PLAN
test
21:55:19 - opendevin:ERROR: llm.py:64 - argument of type 'NoneType' is not iterable. Attempt #1 | You can customize these settings in the configuration.
21:55:21 - opendevin:ERROR: llm.py:64 - argument of type 'NoneType' is not iterable. Attempt #2 | You can customize these settings in the configuration.
21:55:22 - opendevin:ERROR: llm.py:64 - argument of type 'NoneType' is not iterable. Attempt #3 | You can customize these settings in the configuration.
21:55:24 - opendevin:ERROR: llm.py:64 - argument of type 'NoneType' is not iterable. Attempt #4 | You can customize these settings in the configuration.
21:55:30 - opendevin:ERROR: llm.py:64 - argument of type 'NoneType' is not iterable. Attempt #5 | You can customize these settings in the configuration.
21:55:30 - opendevin:ERROR: agent_controller.py:103 - Error in loop
Traceback (most recent call last):
File "/app/.venv/lib/python3.12/site-packages/litellm/main.py", line 736, in completion
optional_params = get_optional_params(
^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/litellm/utils.py", line 4981, in get_optional_params
_check_valid_arg(supported_params=supported_params)
File "/app/.venv/lib/python3.12/site-packages/litellm/utils.py", line 4630, in _check_valid_arg
if k not in supported_params:
^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/app/opendevin/controller/agent_controller.py", line 99, in _run
finished = await self.step(i)
^^^^^^^^^^^^^^^^^^
File "/app/opendevin/controller/agent_controller.py", line 212, in step
action = self.agent.step(self.state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/agenthub/SWE_agent/agent.py", line 90, in step
action, thought = self._think_act(messages=msgs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/agenthub/SWE_agent/agent.py", line 45, in _think_act
resp = self.llm.completion(
^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/tenacity/init.py", line 289, in wrapped_f
return self(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/tenacity/init.py", line 379, in call
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/tenacity/init.py", line 325, in iter
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/tenacity/init.py", line 158, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/app/.venv/lib/python3.12/site-packages/tenacity/init.py", line 382, in call
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/app/opendevin/llm/llm.py", line 79, in wrapper
resp = completion_unwrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/litellm/utils.py", line 2977, in wrapper
raise e
File "/app/.venv/lib/python3.12/site-packages/litellm/utils.py", line 2875, in wrapper
result = original_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/litellm/main.py", line 2137, in completion
raise exception_type(
^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/litellm/utils.py", line 8665, in exception_type
raise e
File "/app/.venv/lib/python3.12/site-packages/litellm/utils.py", line 8640, in exception_type
raise APIConnectionError(
litellm.exceptions.APIConnectionError: argument of type 'NoneType' is not iterable

[gbenaa]

I do realise btw that this is broadly the same issue as #1208

but I still can't seem to find a resolution for that anywhere? Perhaps there is and I just can't find it.

Thanks

[enyst]

FWIW I have seen reports that ollama/llama3 runs successfully - this comment

I'm sorry for the frustration. I'm not aware of an issue preventing you from running, but we can definitely improve the experience here, as we learn what and where the problems are.

The error you get (initially, "argument of type 'NoneType' is not iterable") is not one reported before, as far as I can see. It's also a bit curious, it seems to complain that supported_params is wrong... if I see this right. Can you please try a command in the opendevin directory:

poetry show litellm

What version does it have?

Perhaps more importantly to make it work now, you're running the 'main' opendevin docker image. Can you try 0.4.0 instead of main?

The last one: as we can see in the linked comment, it appears that ollama:llama3 can work. Use that, and, can you please check and tell what ollama list shows?

[gbenaa]

Thanks enyst. It's okay, I'm not frustrated at all :) I know your product's in early alpha, just really keen to get it working :)

Here is the output you were asking for... thanks again.

$ poetry show litellm

RuntimeError

The Poetry configuration is invalid:
- Additional properties are not allowed ('group' was unexpected)

at /usr/lib/python3/dist-packages/poetry/core/factory.py:43 in create_poetry
39│ message = ""
40│ for error in check_result["errors"]:
41│ message += " - {}\n".format(error)
42│
→ 43│ raise RuntimeError("The Poetry configuration is invalid:\n" + message)
44│
45│ # Load package
46│ name = local_config["name"]
47│ version = local_config["version"]

[gbenaa]

I'll set about fixing this, as it's clearly indicative of a problem. Don't worry about commenting again in the meantime, unless you want to.

Thanks :)

[SmartManoj]

RuntimeError

The Poetry configuration is invalid: - Additional properties are not allowed ('group' was unexpected)

Poetry configuration is invalid - Additional properties are not allowed ('group' was unexpected)

--
Run this to check whether LLM is working properly.

import tomllib as toml
from litellm import completion
from datetime import datetime
file_path=r'config.toml'
config = toml.load(open(file_path,'rb'))

messages = [{ "content": "If there are 10 books in a room and I read 2, how many books are still in the room?","role": "user"}]
dt = datetime.now()
response = completion(model=config['LLM_MODEL'], 
                        api_key=config['LLM_API_KEY'],
                        base_url=config.get('LLM_BASE_URL'),
                      messages=messages)

content = response.choices[0].message.content
print(content)

if '8' in content:
    print('--> Correct answer! 🎉')
    print('There are still 10 books in the room; reading them does not reduce the count. Consider exploring more accurate models for better results.')

dt2 = datetime.now()
print('Used model:',config['LLM_MODEL'])
import os; print('AZURE_API_VERSION',os.environ.get('AZURE_API_VERSION'))
print(f"Time taken: {(dt2-dt).total_seconds():.1f}s")