Use private ip with external cloud manager that doesn't know about that ip

[gsimko]

Is your feature request related to a problem? Please describe.

I'm trying to set up wireguard to protect all the communications between my k8s nodes.
I want that to include the initialization steps as well (the script from https://get.k3s.io), so the flannel CNI cannot address this.
I'm doing this in the hetzner cloud, where I cannot use the hetzner private network because it doesn't support connections between different regions and I'm aiming for global coverage.

The problem is that if I set node-ip to the desired private ip, the CCM fails because it won't be able to identify the node by the "alpha.kubernetes.io/provided-node-ip" label. Furthermore, this label is not allowed to be initialized on the kubelet startup, so I cannot change it.

Describe the solution you'd like
It would be ideal to separate the two uses of node-ip and add control to manage them separately:

1. it controls where the kube-apiserver and etcd listens (this is what I'd like to set to my private ip)
2. it controls how to CCM matches the node through "alpha.kubernetes.io/provided-node-ip" (this is where I'd like to keep using the original public ip)

I'm also open for alternative suggestions, but I couldn't figure out any.

[brandond]

the CCM fails because it won't be able to identify the node by the "alpha.kubernetes.io/provided-node-ip" label. Furthermore, this label is not allowed to be initialized on the kubelet startup, so I cannot change it.

Are you referring to the hetzner CCM? The k3s embedded CCM doesn't rely on this label for anything. If you're fighting with the hetzner cloud provider, you might consider opening an issue on their repo, instead of here?

[gsimko]

I think hetzner CCM does the right thing with the label based on https://kubernetes.io/docs/reference/labels-annotations-taints:
"This IP is verified with the cloud provider as valid by the cloud-controller-manager."

So according to this the provided-node-ip label should be set to the external IP, but it would be still nice to have the nodes communicate with each other by their private IPs.

[gsimko]

FWIW it's not even the Hetzner CCM that fails but the following k8s code: https://github.com/kubernetes/cloud-provider/blob/master/node/helpers/address.go#L105