Not able to run docker exec with both Ollama (llama3) and Docker installed on WSL2 (ubuntu). What am I missing?

[tejeshwar-p]

Describe your question

I have done setup of both Ollama (llama3) and Docker in my WSL2 (ubuntu). My operating system is Windows 10.
I get success response from model with these commands when I only test Ollama -
ollama list and curl http://localhost:11434/api/generate -d '{"model":"llama3","prompt":"hi"}'

However when I run this command -
docker exec f5f7a105ee5f curl http://host.docker.internal:11434/api/generate -d '{"model":"llama3","prompt":"hi"}'
I always get the same error -
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (7) Failed to connect to host.docker.internal port 11434 after 0 ms: Couldn't connect to server

Can someone please help as no matter what I do, the setup just does not work with ollama and docker both installed in WSL?

Additional context

I have followed all the steps as mentioned in - https://github.com/OpenDevin/OpenDevin/blob/main/docs/guides/LocalLLMs.md#4-configuring-the-ollama-service-wsl

I have installed docker latest version and also followed the post installation steps of docker to run without requirement of sudo as described in - https://docs.docker.com/engine/install/linux-postinstall

Export command -
export WORKSPACE_BASE=$(pwd)
My docker run command -

docker run \
--add-host=host.docker.internal:host-gateway \
    -e LLM_API_KEY="ollama" \
    -e LLM_BASE_URL="http://host.docker.internal:11434" \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    ghcr.io/opendevin/opendevin:0.3.1

FYI, I tried with --add-host host.docker.internal=host-gateway as well but still same issue.
I have shutdown all the WSL2 sessions and restarted WSL2 but still no luck.

[COWI-MI]

Failed to connect to host.docker.internal port 11434 after 0 ms: Couldn't connect to server

It looks like a Docker or host problem. Not something related to OpenDevin.

[tejeshwar-p]

Failed to connect to host.docker.internal port 11434 after 0 ms: Couldn't connect to server

It looks like a Docker or host problem. Not something related to OpenDevin.

Yes. That's why I added as question. Not issue directly related to OpenDevin.
I want to understand if there is any other configuration which is missing.
I followed the exact steps mentioned in the documentation but not able to get it work with ollama and docker installed in WSL2. If someone can try out the same setup it will be helpful to answer this question.

I have some rough steps which I noted when I setup my ollama and docker. I can share those here if it helps.

[tejeshwar-p]

Steps overview -

1. Install WSL2 for windows 10
2. Install Ubuntu 2204
3. Install Ollama in WSL2
4. Install Docker in WSL2
5. Install OpenDevin in WSL2
6. Run OpenDevin in WSL2

Here are detailed 3, 4 and 6 steps -

3. Download and install ollama in ubuntu

Commands -
a. curl -fsSL https://ollama.com/install.sh | sh

b. ollama serve

c. curl http://127.0.0.1:11434/

d. ollama run llama3

4. Install Docker

Steps from this official docker link ->
Install Docker Engine on Ubuntu | Docker Docs

Install using the apt repository
Before you install Docker Engine for the first time on a new host machine, you need to set up the Docker repository. Afterward, you can install and update Docker from the repository.

Set up Docker's apt repository.

Add Docker's official GPG key:

• sudo apt-get update
• sudo apt-get install ca-certificates curl
• sudo install -m 0755 -d /etc/apt/keyrings
• sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
• sudo chmod a+r /etc/apt/keyrings/docker.asc

Add the repository to Apt sources:

• echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

• sudo apt-get update

Note
If you use an Ubuntu derivative distro, such as Linux Mint, you may need to use UBUNTU_CODENAME instead of VERSION_CODENAME.

Install the Docker packages.

To install the latest version, run:
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify that the Docker Engine installation is successful by running the hello-world image.

sudo docker run hello-world

This command downloads a test image and runs it in a container. When the container runs, it prints a confirmation message and exits. You have now successfully installed and started Docker Engine.

Tip
Receiving errors when trying to run without root?
The docker user group exists but contains no users, which is why you’re required to use sudo to run Docker commands. Continue to Linux postinstall to allow non-privileged users to run Docker commands and for other optional configuration steps.

Post install steps

Post install steps official link -> Linux post-installation steps for Docker Engine | Docker Docs

• Manage Docker as a non-root user by creating a Docker group and adding your user to it:
  sudo groupadd docker
sudo usermod -aG docker $USER
  You’ll need to log out and back in for this to take effect, or you can run the following command to activate the changes in the current session: newgrp docker

• Configure Docker to start on boot. For systems with systemd support, you can enable Docker to start on boot using:
  sudo systemctl enable docker.service
sudo systemctl enable containerd.service
  If your WSL does not support systemd, you can add the Docker service start command to your shell’s profile script (like ~/.bashrc or ~/.zshrc).

• Test your Docker installation by running: docker run hello-world

6. Run OpenDevin

• export WORKSPACE_BASE=$(pwd)

• Run either of the below commands -

docker run \
    --add-host host.docker.internal=host-gateway \
    -e LLM_API_KEY="ollama" \
    -e LLM_BASE_URL="http://host.docker.internal:11434" \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    ghcr.io/opendevin/opendevin:0.3.1

OR

docker run \
--add-host=host.docker.internal:host-gateway \
    -e LLM_API_KEY="ollama" \
    -e LLM_BASE_URL="http://host.docker.internal:11434" \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    ghcr.io/opendevin/opendevin:0.3.1

Validation commands for ollama and docker connectivitity

• ollama list # get list of installed models
• curl http://localhost:11434/api/generate -d '{"model":"llama3","prompt":"hi"}' # check if ollama is accessible
• docker ps # get list of running docker containers, for most accurate test choose the open devin sandbox container.
• docker exec [CONTAINER ID] curl http://host.docker.internal:11434/api/generate -d '{"model":"[NAME]","prompt":"hi"}' # this command fails with correct docker id also -> Need help here please!

[zhonggegege]

It's more suitable for your docker to be installed on the windows host. You can try it. WSL does't require additional docker.Because "host.docker.internal:host-gateway" is to solve the communication between windows host and wsl network.

[enyst]

@tejeshwar-p Thank you for the detailed steps. Can you please confirm, have you also tried section Fixing It: https://github.com/OpenDevin/OpenDevin/blob/main/docs/guides/LocalLLMs.md#fixing-it ?

[tejeshwar-p]

@tejeshwar-p Thank you for the detailed steps. Can you please confirm, have you also tried section Fixing It: https://github.com/OpenDevin/OpenDevin/blob/main/docs/guides/LocalLLMs.md#fixing-it ?

Hi @enyst - Yeah. Already did that. This is how my ollama.service file looks -

[SmartManoj]

https://forums.docker.com/t/connection-refused-on-host-docker-internal/136925/6

[tejeshwar-p]

https://forums.docker.com/t/connection-refused-on-host-docker-internal/136925/6

Thank you @SmartManoj. Will try and keep result posted. So, it means I have to do some tinkering with the existing OpenDevin's entrpoint.sh and Dockerfile.