Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Watchtower failed to create containers on host network #1906

Open
c-def opened this issue Jan 20, 2024 · 30 comments
Open

Watchtower failed to create containers on host network #1906

c-def opened this issue Jan 20, 2024 · 30 comments

Comments

@c-def
Copy link

c-def commented Jan 20, 2024

Describe the bug

Watchguard has been keeping my containers up to date for years. Today I installed an update to Docker, and when Watchtower attempted to update two of my containers on the host network, it failed with the following error:

invalid config for network host: invalid endpoint settings: network-scoped alias is supported only for containers in user defined networks

These were both created via docker-compose with network_mode set to host

Steps to reproduce

Make Watchtower attempt to update a container on the host network.

Expected behavior

New container is created successfully.

Screenshots

No response

Environment

  • Platform: Ubuntu
  • Architecture: 64 bit
  • Docker version 25.0.0, build e758fe5

Your logs

ERRO[12591] Error response from daemon: invalid config for network host: invalid endpoint settings:
network-scoped alias is supported only for containers in user defined networks 
INFO[12591] Creating /sonarr                             
INFO[12595] Creating /home-assistant                     
ERRO[12595] Error response from daemon: invalid config for network host: invalid endpoint settings:
network-scoped alias is supported only for containers in user defined networks


Here is the log from one week ago when it worked just fine:
2024-01-13T10:02:31.051969197Z INFO[3032453] Creating /home-assistant                     
2024-01-13T10:02:34.702164592Z INFO[3032457] Creating /sonarr

Additional context

No response

Copy link

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

@rany2
Copy link

rany2 commented Jan 20, 2024

I'm also facing the same issue:

time="2024-01-20T18:16:36Z" level=info msg="Creating /container1"
time="2024-01-20T18:16:36Z" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-20T18:16:36Z" level=info msg="Creating /container2"
time="2024-01-20T18:16:36Z" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"

@jgeusebroek
Copy link

It also happens with non-host networking.

@manderss99
Copy link

I also have this with containers that use both host and bridge network, started recently, possibly by docker-ce update.
docker-ce/jammy,now 5:25.0.0-1ubuntu.22.04jammy

@jmnovak50
Copy link

Seeing also with Raspberry Pi
Docker version 25.0.0, build e758fe5

@BrodyBuster
Copy link

I also have this with containers that use both host and bridge network, started recently, possibly by docker-ce update. docker-ce/jammy,now 5:25.0.0-1ubuntu.22.04jammy

Debian Bookworm - Docker version 25.0.0, build e758fe5

Being there hasn't been an update to watchtower in some time, I also suspect this is a docker issue. Following along here, in the event a solution is found. For the time being I had to disable watchtower.

@MrEAlderson
Copy link

Same problem. What's worse is that it suddenly deleted the whole container without recreating it...

@alexandervnuchkov
Copy link

In my instance, this affects containers created using docker run. Containers started with docker compose update without any problem. The issues started after docker updated to version 25.0.0

@rany2
Copy link

rany2 commented Jan 21, 2024

Containers started with docker compose update without any problem.

Actually, my containers are all started with docker compose and had this problem.

@c-def
Copy link
Author

c-def commented Jan 21, 2024

Just wanted to mention I have the same issue with Portainer if I try to edit/duplicate these same containers.

@farces
Copy link

farces commented Jan 22, 2024

Chiming in as well, created using docker-compose and on the bridge network. There was a docker-ce update a few days ago so it's entirely possible that's the culprit, but the first symptom I had was the error during watchtower recreating the container.

Starting manually in Portainer worked fine afterwards.

time="2024-01-22T00:07:24Z" level=info msg="Found new jwilder/nginx-proxy:latest image (c55e0435bcab)"
time="2024-01-22T00:07:25Z" level=info msg="Stopping /nginx-proxy (b23e7978ed10) with SIGTERM"
time="2024-01-22T00:07:35Z" level=info msg="Creating /nginx-proxy"
time="2024-01-22T00:07:35Z" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-22T00:07:35Z" level=info msg="Session done" Failed=1 Scanned=7 Updated=0 notify=no

Docker version details:

face@localhost:~/nginx-proxy$ docker version
Client: Docker Engine - Community
 Version:           25.0.0
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:49 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       615dfdf
  Built:            Thu Jan 18 17:09:49 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

@jmnovak50
Copy link

jmnovak50 commented Jan 22, 2024

Not sure this has something to do with it. This also failed for a container being configured on the system defined bridge network...Created a non-system created bridge network and then recreated the container under that user-defined bridge network. An update came in for that container and it recreated just fine. Wondering if this has to do with the system defined networks (bridge, host, none) specifically? Worth noting I also moved the Watchtower container to the user defined bridge network.

@CJO100293
Copy link

Just chiming in to say ive also got this same problem. Started around the same time as everyone else.

@rursache
Copy link

rursache commented Jan 22, 2024

same issue here, watchtower kills and delete the containers but never re-creates them. as someone with over 100 containers, playing "what disappeared last night" everyday is very annoying. this bug should be highest priority not only "medium"

time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /immich"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /privatebin"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /fluidd"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /calibre-web"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /yourls"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /vscodeserver"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /sonarr"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /pairdrop"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /homepage"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"

seems to be happening since updating docker to 25.0.0

@spupuz
Copy link

spupuz commented Jan 22, 2024

same error and problem here

@BrodyBuster
Copy link

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

@xinud190
Copy link

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

Can you recover containers Watchdog had stopped and creating (but received this error) ?

Can you make the change you mentioned and this will allow them to recover / watchdog complete the creation?

@spupuz
Copy link

spupuz commented Jan 22, 2024

this is happening not only on host container but also on container already configured with bridge, and this is random since container in bridge are not recrarete but at the next update after a compose restart they update work properly.

@BrodyBuster
Copy link

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

Can you recover containers Watchdog had stopped and creating (but received this error) ?

Can you make the change you mentioned and this will allow them to recover / watchdog complete the creation?

When my containers failed, prior to moving them to a user bridge, the updated image was downloaded, the container was stopped and deleted, and watchtower failed to recreate the container and start it. I had recreate the container using docker run / docker compose.

This morning I moved containers that had failed previously on the default host network to a user bridge, with the exception of one container. All of these containers had available updates. The bridge containers updated fine, the host network container failed.

I will continue to monitor this to see if the bridge containers end up failing gain.

@rmtsrc
Copy link

rmtsrc commented Jan 22, 2024

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

I'm facing the same problem, however some containers need access to my host network, so switching them to a bridged network wouldn't work for me.

@CJO100293
Copy link

CJO100293 commented Jan 22, 2024

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

Can you recover containers Watchdog had stopped and creating (but received this error) ?
Can you make the change you mentioned and this will allow them to recover / watchdog complete the creation?

When my containers failed, prior to moving them to a user bridge, the updated image was downloaded, the container was stopped and deleted, and watchtower failed to recreate the container and start it. I had recreate the container using docker run / docker compose.

This morning I moved containers that had failed previously on the default host network to a user bridge, with the exception of one container. All of these containers had available updates. The bridge containers updated fine, the host network container failed.

I will continue to monitor this to see if the bridge containers end up failing gain.

Unfortunately ive got several containers that need access to the host network, so like rmtsrc, using bridge network unfortunately isnt a fix.

@etho201
Copy link

etho201 commented Jan 22, 2024

Many containers supporting the home-assistant ecosystem also require the containers to share its network namespace with the host machine. These have been failing when updated using Watchtower; however, Docker compose is able to bring them up without issue. Hoping the issue with Watchtower is resolved soon!

@xinud190
Copy link

I ended up restoring my Proxmox VM Backup of the Linux VM I'm using running docker. Restored all my containers and I disabled Watchtower so it doesn't upgrade containers to latest image.

Hoping this gets resolved soon.

@3shirts
Copy link

3shirts commented Jan 23, 2024

I have this too but it doesn't seem to be exclusive to Watchtower. If I manually try to redeploy a container from Portainer I get this same error. Redeploying the stack works though so docker compose is the workaround.
Looks like a docker-related change that's caused this.

@alfchao
Copy link

alfchao commented Jan 23, 2024

msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"

@xinud190
Copy link

Looks like they are fixing this in docker ...

docker/for-linux#1481 (comment)

@ptr727
Copy link

ptr727 commented Jan 24, 2024

Same here:
time="2024-01-22T02:03:26-08:00" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"

@xinud190
Copy link

xinud190 commented Jan 24, 2024

FYI - Problem for docker was resolved and fixes released

docker/for-linux#1481 (comment)

docker/for-linux#1481

Personally, I will be waiting a bit before updating docker and re-enabling Watchtower.

If anyone else tests updated docker with Watchtower to confirm its fixed love to hear it.

@CJO100293
Copy link

FYI - Problem for docker was resolved and fixes released

docker/for-linux#1481 (comment)

docker/for-linux#1481

Personally, I will be waiting a bit before updating docker and re-enabling Watchtower.

If anyone else tests updated docker with Watchtower to confirm its fixed love to hear it.

Ive tested and confirmed that the newest update to docker-ce (25.0.1) has resolved the above issues for me

@farces
Copy link

farces commented Jan 25, 2024

Can also confirm 25.0.1 of docker-ce resolves the issue. I think this can be closed as it’s not a watchtower specific issue regardless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests