How to setup k3s Cluster with disabled external Interfaces

[derlinuxer]

Some pre-condition:

• hetzner-k3s version >= 1.1.4 needed
• create private network in your hetzner project and use the name in cluster config existing_network
• I use a pfsense image in the private network to NAT the trafic from cluster into the world (other solutions possible)
• create a static route in hetzner private network: 0.0.0.0 -> pfSense-IP (or other NAT instance)
  • in this case we have also a fixed outgoing IP which is need for use to allow connections to customer systems to open there FW.

Example config for test cluster:

hetzner_token: <your token>
cluster_name: rke2-test
kubeconfig_path: "./kubeconfig"
k3s_version: v1.26.5+k3s1
public_ssh_key_path: "~/.ssh/id_rsa.pub"
private_ssh_key_path: "~/.ssh/id_rsa"
use_ssh_agent: false
ssh_allowed_networks:
  - 0.0.0.0/0
api_allowed_networks:
  - 0.0.0.0/0
schedule_workloads_on_masters: false
private_network_subnet: 10.2.0.0/16
existing_network: net-kube-test
enable_public_net_ipv4: false
enable_public_net_ipv6: false
disable_flannel: false
cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.16.0/ccm-networks.yaml"
system_upgrade_controller_manifest_url: "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"
csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.3.2/deploy/kubernetes/hcloud-csi.yml"
masters_pool:
  instance_type: cax11
  instance_count: 1
  location: fsn1
  image: 103908130
worker_node_pools:
- name: arm
  instance_type: cax21
  instance_count: 1
  location: fsn1
  image: 103908130
- name: x86
  instance_type: cx21
  instance_count: 1
  location: fsn1
  post_create_commands:
    - timedatectl set-timezone Europe/Berlin
    - >
      printf "network## {config## disabled}" |
      sed 's/##/:/g' > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
    - >
      printf "network##\n  version## 2\n  renderer## networkd\n  ethernets##\n    ens10##\n      dhcp4## true\n      nameservers##\n        addresses## [185.12.64.1, 185.12.64.2]\n      routes##\n        - to## default\n          via## 10.2.0.1" |
      sed 's/##/:/g' > /etc/netplan/50-cloud-init.yaml
    - netplan generate
    - netplan apply
    - apt update
    - apt upgrade -y
    - apt install nfs-common -y
    - apt autoremove -y
post_create_commands:
  - timedatectl set-timezone Europe/Berlin
  - >
    printf "network## {config## disabled}" |
    sed 's/##/:/g' > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
  - >
    printf "network##\n  version## 2\n  renderer## networkd\n  ethernets##\n    enp7s0##\n      dhcp4## true\n      nameservers##\n        addresses## [185.12.64.1, 185.12.64.2]\n      routes##\n        - to## default\n          via## 10.2.0.1" |
    sed 's/##/:/g' > /etc/netplan/50-cloud-init.yaml
  - netplan generate
  - netplan apply
  - apt update
  - apt upgrade -y
  - apt install nfs-common -y
  - apt autoremove -y

Some explanations:

• I run hetzner-k3s on a small machine inside the hetzner private network (also possible with configured VPN on your local machine)
• post_create_commands for cloud init will create own network configuration on VM creation
  • two section of it: one global for ARM Pools master and worker-arm, second for worker-x86
  • this is needed because of different interface names
• the sed command after printf is a bit confusing but if I set the : directly the yaml parser crashes. So I use ## and replace it the end. Didn't found anythink to mask the : (Maybe you found it ;-))
• you have to be carefull this spaces for the yaml format in netconfig file
• the corresponding netconfig file looks like this:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp7s0:
      dhcp4: true
      nameservers:
        addresses: [185.12.64.1, 185.12.64.2]
      routes:
        - to: default
          via: 10.2.0.1

Deployment grafik

Small graphic to get an overview (Network is different to example config):

[vitobotta]

To clarify, the tool installs k3s, not rke2 :)

[mgalesloot]

Successfully tested the derlinuxer script with a NAT Gateway as per https://community.hetzner.com/tutorials/how-to-set-up-nat-for-cloud-networks. I used a bastion host in the private network from where I could run the script and access the private hosts.

[derlinuxer]

Nice that it's working. :-) And thank for the link to setup a clean and easy NAT GW.

[easystartup-io]

Have verified this it works for me by setting up a new cluster along with the above provided nat gateway setup. Just one issue is when its a three node HA master cluster, the k8s api load balancer ip is still exposed to the public. Need to verify if the merged pr of @mgalesloot will help make the lb private and isolate all incoming connections to only the exposed application level load balancers.

[easystartup-io]

Do note that the interface need not be ens10 for x86. Based on the instance type the network interface can also be enp7s0 as mentioned in the docs here. https://docs.hetzner.com/cloud/networks/server-configuration/#debian--ubuntu . You can ssh into your master/worker node and run ip a to find all attached interfaces.

[mgalesloot]

Try to access the k8s api via the private IP of the LB. This should work. Next you can disable public access of the hetzner LB.
You can make it more fancy by setting a hostname for the LB, but this is not really required in order to make the cluster api private.

[easystartup-io]

Disabling the public network and manually editing kubeconfig file stops the worker nodes to be able to connect to the master and this also fails hetzner-k3s create --config cluster_config.yaml, so I need to try the hostname method

[easystartup-io]

hostname is working perfectly, an A record to the private ip of the load balancer is working perfectly as expected, thanks for your pr. I suppose setting the api_server_hostname directly to the private IP of the LB also might make it behave in exactly same manner. But I am sticking to DNS resolution for now.

[jampy]

Since the pfsense host is a potential single point of failure I guess this setup is not ideal for clusters (applications) that need to contact a lot of external APIs, am I right?

Would it be an alternative to block all incoming connections on the nodes using the Hetzner firewall that's available on the cloud servers and let the nodes talk to external hosts directly?

[pgruener]

As I noted in the past, the point is, that the hetzner workers cannot communicate to external hosts, when they dont have external IPs assigned. So if the workers should be kept only in an internal network, some NAT is necessary for them to talk to external hosts.

So as the pfSense is a single point of failure, it probably needs to be HA as well.

What I am considering is, if it would make sense to give Public IPs to ingress nodes (which could have ext ips) only and make them the NATs for the Workers.
What concerns what be against that structure?