Single watchtower instance for multiple remote hosts?

[rwjack]

Is your feature request related to a problem? Please describe.

I've come across an interesting use case where it would be useful to have a single watchtower instance that collects information about all the deployed images across the environment (all the monitored remote docker hosts), and pulls new versions to the watchtower host itself.

From there, those images can be scanned for vulnerabilities and deployed if a pipeline passes. That way the only service requiring internet access for image repositories would be the watchtower host itself.

This would also require having host based lifecycle hooks, which would not execute within the remote running containers, but on the watchtower host itself. Something like when a new image is found: curl gitlab/api/scan&deploy/pipeline/new

Describe the solution you'd like

^

Describe alternatives you've considered

^

Additional context

No response

[rwjack]

This gives watchtower an actual business use case, acting as a gatekeeper for secure image deployment, rather than just being a fire&forget image updater.

[MarkDPierce]

Outside the stuff not exactly the intention of watchtowers use case. There is capability to run and update remote hosts https://containrrr.dev/watchtower/remote-hosts/

[rwjack]

That would mean 1 watchtower instance = 1 remote host. Also, the image pull would still be performed on the remote host.

I'm working on a pipeline (gitlab-ci + ansible) that would pull images directly to the runner, perform security scans there, and from there deploy to the other docker hosts. This means my hosts don't need access to dockerhub, quay, nor do they need keys for private registries.

The last missing piece of the puzzle - Having a master watchtower instance, with a web ui, which pulls images on a schedule to that master host, and shows the admin a visual representation of images that are to be deployed, images that were recently pulled, image SHA history, etc.

[MarkDPierce]

Oh. We solve this by shipping 'good' images from their source to a registry we own in Google Cloud and point watchtower to that. No need to have yet another scanning and validating tool since all public registries now provide this information.

We use 1 single instance of watchtower to control a fleet of 10 servers.

[rwjack]

Interesting. How do you "ship" images from their source to your registry?

And by pointing watchtower to your registry, what do you mean by that? Doesn't watchtower need to look at actively deployed images/containers?

As for the security scans, yes, most images have that information public by now, but I prefer having security policies checked at deploy time, rather than at "ship" time. By "ship", I mean from your terminology, when the image is passed to the internal registry.

Also, how can you control 10 servers with 1 watchtower instance?

[MarkDPierce]

https://containrrr.dev/watchtower/private-registries/ For pointing it to Google Artifact registry and we're using https://github.com/tammert/replicant to sync our containers but we're migrating to a more open solution in Crane or GCrane https://github.com/google/go-containerregistry. So we have a bit of custom Golang that looks at GCP security center and based on some requirements we will flag "safe" images and we ship those to a 'pull' registry with an included latest tag. Watchtower is pointed to this pull registry as a source of truth.

And sorry, to clarify we use a single host to host multiple watchtower instances for their respective hosts. Since the footprint is low this is not a problem for us, and in fact, if you use something like a self hosted github style tooling with actions, you can even run a compose file with all the watchtower settings on the runner. Lot of ways to skin this cat though.

[rwjack]

Very interesting solution indeed. Thanks for the clarification!