Deploying a "pure IPv6" cluster (without public IPv4 addresses)

[jpetazzo]

In theory, it should be possible to deploy a Kubernetes cluster without public IPv4 addresses. There would be no significant loss of functionality for the users, since the API server is exposed with a load balancer; and public-facing services running inside the Kubernetes cluster could be exposed with type: LoadBalancer.

In practice, there are multiple issues to address if we want to achieve that with hetzner-k3s.

I'm sharing my experimentations and thoughts here in case that's helpful to others.

1. Connecting to the nodes

hetzner-k3s executes the provisioning by connecting to the nodes over SSH. As of April 2024, it uses the public IPv4 address of the nodes, and if the nodes don't have a public IPv4 address, it falls back to their private address. This only works if hetzner-k3s is executed on a machine connected to the same Hetzner Private Network as the cluster nodes. I imagine that it might be possible to create the Private Network ahead of time; create a Hetzner VM there; and run hetzner-k3s from that machine; but I didn't try (I wanted to be able to provision clusters from my local machine).

The idea would be to do the provisioning over IPv6. Of course, it requires to have a machine with IPv6 connectivity; the rest of this document assumes that you have IPv6 connectivity. It's the case with all the ISP I've used in the last 5+ years in France, Germany, and the US. It might not be the case when connecting from hotels, conferences, hotspots, etc., though.

I patched hetzner-k3s so that when there is no public IPv4 address, it falls back to the public IPv6 address (and if there is no public address at all, it falls back to the private IPv4 address, like before). The patch is available in my ipv6-WIP branch - but it's not enough to solve the problem.

2. Determining the public IP address

Currently, the public IP address is determined with hostname -I - more precisely, PUBLIC_IP=$(hostname -I | awk '{print $1}'). This gives the first IPv4 address found on the machine. When the nodes have a public IPv4 address, it yields the correct result; but when the nodes don't have a public IPv4 address, it yield an address in the carrier-grade private IP space (10.64.0.0/10).

There are multiple possibilities to find out the public IPv6 address; for instance, PUBLIC_IP=$(hostname -I | awk '{print $3}') seems to work (but it might break if Hetzner changes the way their configure their VMs).

3. Downloading k3s

When Hetzner machines don't have a public IPv4 address, they can't access external servers that have IPv4-only connectivity. And, unfortunately, github.com is V4-only. So when the k3s install script tries to download k3s hashes or k3s binaries, that fails.

I'm stuck at this point. I could imagine the following solutions:

• hotfix the k3s install script (e.g. with sed) to update github.com URLs with a mirror that would be reachable on IPv6
• deploy an extra node with an IPv4 public address and use it as a NAT gateway or HTTP proxy for the other nodes
• deploy the nodes with a public IPv4 address, and once everything is up and running, remove the IPv4 address

None of these solutions is truly satisfactory. The "least worst" might be the dual stack gateway/proxy, but if someone has a better idea, let me know :)

4. ??? (the unknown unknowns!)

After we find a way to download k3s, we'll probably run into other issues (maybe with Flannel or otherwise). I have experience running full-IPv6 clusters on Linode with Cilium, so I'm hopeful; but we'll see :)

[vitobotta]

Thanks for looking into this. To be honest I haven't seen a case where I ever needed to disable ipv4 because at this point in time it can still translate into hassle for a number of things. Besides saving on the cost of IPv4 addresses, what is the use case?

[jpetazzo]

It's mostly "preparedness" - i.e., I think we should be forward-thinking and stop using public IPv4 addresses where we can. A Kubernetes cluster is the perfect example where public IPv4 addresses shouldn't be necessary. I've even built a few "pure IPv6" clusters, and it's very interesting: pods get public IPv6 addresses, and all services are automatically public-facing because the Cluster IP CIDR is a public IPv6 block.

Also, when I saw that there were options to disable IPv4 public addresses, I thought "whoa, hetzner-k3s supports this, that's awesome" so I wanted to test it. After trying it, I wonder if anyone is actually using that option; perhaps we should add something in the README/doc to warn people that it doesn't work out of the box 😅

[vitobotta]

Yeah that would be a good idea. To be honest I haven't done much testing without public IPv4 addresses. It was a PR, I did a little testing then and I basically forgot about it afterwards since it hasn't come up again until you started this discussion :)

I agreed that it's a good idea to be prepared and address this. I just have other things on the list (https://github.com/users/vitobotta/projects/4) so this perhaps isn't a high priority yet. But if you have a chance to look into it I always welcome PRs! :)

[jpetazzo]

Absolutely! I should have indicated in my original message that this was purely "research" - I wanted to gather informations about what worked and what didn't just in case others were interested. But I agree that there is no strong business need there :)

[adrianalter]

What about without both Ipv4 and ipv6? And using a gateway for security reasons?

[adrianalter]

I'm thinking having (example) 3 masters 5 workers without ipv4 ipv6 and then one server that acts as gateway that has ipv4 and can connect to the internet. I have 4 VMs on AWS without ipv4/6 and they go through an internet gateway to connect with the internet, all residing on a single VPC.
If we could have the same thing on Hetzner it would render AWS useless :D

Am i missing something or got anything wrong though? xD

[adrianalter]

@jpetazzo just saw your comment pop up after i replied lol! I'll try to play around and see if I can come up with a reasonable flow to get it done, in case i'll document it

[vitobotta]

I see. We actually do that at work with GKE, in that all our clusters in Google Cloud are "private" and totally disconnected from the public Internet so that direct attacks against the clusters are not possible. But there is a difference from the typical setup in Hetzner, hence my questions earlier. To access the GKE clusters remotely, we use bastion hosts that are also private and not directly accessible from the Internet. So to access these clusters we Google's Identity Aware Proxy to establish an SSH tunnel to the bastion host, and then use tinyproxy on the bastion host to proxy the Kubernetes API. it's all transparent to the user thanks to some tooling that I built but this is how it works behind the scenes. Max security but in this case we have the Identity Aware Proxy, which we don't have with Hetzner. This is why I was asking those questions.

Even if you access the Hetzner clusteres via a bastion host in the same private nerwork, the bastion host still needs a public interface for you to connect remotely to it, right? I guess you can restrict connections to it via firewall but it's not the same thing as with IAP.

But yes, generally speaking what @jpetazzo was suggesting is what I'd try too.

[jpetazzo]

I gave it a try, and managed to get a cluster without public IPv4 and IPv6. Then I started to write down instructions, and noticed that this had already been done a long time ago 🤦

@adrianalter if you're interested, check out #252 - it explains how to do it. I ended doing it with a slightly different approach but the end result is the same. Happy to share you my notes if that helps.

[vitobotta]

@jpetazzo would you have a chance to consolidate that page and what you were writing into a single coherent document? It would be nice if you could do a PR having the more-refactoring branch as base.