My home Kubernetes cluster ⛵

... managed by Flux and serviced with RenovateBot 🤖

---

📖  Overview

This repository is my home Kubernetes cluster in a declarative state. Flux watches my cluster folder and makes the changes to my cluster based on the YAML manifests.

Feel free to open a Github issue or join the k8s@home Discord if you have any questions.

This repository is built off the k8s-at-home/template-cluster-k3s repository.

---

✨  Cluster setup

This cluster consists of both VMs provisioned on PVE with and without the Terraform Proxmox provider as well as physical nodes. These run k3s provisioned overtop Ubuntu 20.04 using the Ansible galaxy role ansible-role-k3s. This is a semi hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a separate server for (NFS) file storage.

See my server/ansible directory for my playbooks and roles, and server/terraform for infrastructure provisioning.

🎨  Cluster components

• metallb: Provides a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.
• calico: For internal cluster networking.
• traefik: Provides ingress cluster services.
• rook-ceph: Provides persistent volumes, allowing any application to consume RBD block storage.
• SOPS: Encrypts secrets which is safe to store - even to a public repository.
• cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.

---

📂  Repository structure

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

• base directory is the entrypoint to Flux
• crds directory contains custom resource definitions (CRDs) that need to exist globally in my cluster before anything else exists
• core directory (depends on crds) are important infrastructure applications (grouped by namespace) that should never be pruned by Flux
• apps directory (depends on core) is where my common applications (grouped by namespace) are placed.

cluster
├── apps
├── base
├── core
└── crds

🔧  Tools

Tool	Purpose
ansible	Preparing Ubuntu for Kubernetes and installing k3s
flux	Operator that manages your k8s cluster based on your Git repository
go-task	A task runner / simpler Make alternative written in Go
dir-env	Sets environment variable based on present working directory
sops	Encrypts k8s secrets with GnuPG

💻 Nodes

Node	Hostname	CPU	RAM	Storage	Function	Operating System
Lenovo M72e	k8s-master-lt9	2C4T i5 3470T	8GB	180GB SSD	Kube Master Node	Ubuntu 20.04.3 LTS
Lenovo M72e	k8s-master-lx0	2C4T i5 3470T	8GB	180GB SSD	Kube Master Node	Ubuntu 20.04.3 LTS
Lenovo M72e	k8s-master-lr3	2C4T i5 3470T	8GB	180GB SSD	Kube Master Node	Ubuntu 20.04.3 LTS
Proxmox VM w/o GPU passthrough	k8s-w-pve	6C i5 10400	16GB	120GB SSD OS, 350GB NVME ceph virtual block storage	Kube Worker Node	Ubuntu 20.04.3 LTS
Proxmox VM w/ Intel iGPU passthrough	k8s-w-pvegpu	4C i7 8700k	12GB	120GB SSD OS, 350GB NVME ceph virtual block storage	Kube Worker Node	Ubuntu 20.04.3 LTS
HP590	k8s-w-590	4C i3 10100	16GB	500GB SSD OS, 500GB NVME ceph	Kube Worker Node	Ubuntu 20.04.3 LTS

💻 Networking and Virtualization Hosts

PC	Hostname	CPU	RAM	Storage	Function	Operating System
Unraid on Proxmox pve	NAS	8C i5 10400	8GB	3 x 8TB HDD, 3TB HDD, 2 x 240GB SSD	NFS Server	unRaid
Lenovo 310s	pfsense	4C j4205	8GB	100GB SSD	Router	pfsense
Node 304 NAS/Virtualization Host	pve	i5 10400	64GB	500GB SSD, 500GB NVME	Virtualization host	Proxmox 7
Fractal Design NR200 Virtualization Host	pve8700k	i7 8700k	32GB	500GB SSD, 1TB NVME	Virtualization host	Proxmox 7

---

🤝  Thanks

A lot of inspiration for my cluster came from the people that have shared their clusters over at kubesearch.dev