My homelab past HA Kubernetes setup. What lies beyond?

homelabkubernetesdocker

Notice: All following words with their typos and incoherent train of thought has been written without AI assistance.

In a way, my current homelab state is exactly what I had dreamed of years ago:

  • Runs on Kubernetes
  • Has proper SSL setup
  • Instances are running immutable Talos Linux
  • Has decent ingress setup - DMZ separated instances for private and public services
  • Renovate managed applications synced by ArgoCD
  • Replicated, distributed storage powered by Longhorn
  • Alerting via Prometheus/Alertmanager and dashboards with Grafana
  • Highly Available

I worked with Kubernetes on and off for many years and had my CKA renewed twice. I know my way around. When nginx ingress controller was deprecated, I happily ran Cilium Gateway API.

How I got here

I spent countless hours tuning and tweaking, and experimenting with my setup. I used to spend vacation days troubleshooting and trying new approaches to ingress, bare-metal vs. VM tests, storage replication, and so on.

I started with K3S on NixOS. This was simple, but updating NixOS was somewhat stresful. Some packages had to be compiled which would render the node unusable for the time being. I also had control plane node hosting workloads and if etcd was occasionally starving for IOPS and would flip the node to NotReady state. I moved to Talos. My control plane VMs were running on cloud VPS, with worker nodes bare-metal at home. They communicaetd via KubeSpan which worked really well, apart from Prometheus not being able to reach some hosts via their internal IPs.

This setup had been an intermediate to my current setup:

  • Incus on Debian hypervisor with XanMod kernel
  • 3xThinkcentre m920q 8500T modded with 10Gbit Intel cards in PCIe slot, 128G boot SSD in wifi slot and 2TB NVMe for data
  • 1xThinkcentre m90q 14700T (same mods)
  • K8S control plane VM instances and 2 virtualized worker instances
  • One bare-metal node with N100 as a worker
  • One bare-metal node with N100 as ingress controller for public / private ingress
  • Backups to my NAS and offsite S3 compat storage
  • Tight VLAN isolation for ingress, private services and public services

To allow my users from overseas I also host 1 fly machine for fast edge connectivity, which connects directly to my public ingress. It’s a pretty cool setup and I am really proud of it.

So it’s all good. My homelab is done. Right?

Now, my setup runs on auto-pilot, and largely unattended. These days I only merge Renovate PRs.

I learned to rely on my services for daily tasks. Paperless to manage invoices and manuals. Changedetection for price tracking on things I need/want, but only at the right price. Immich for sharing my wedding photos with friends and family. It’s a typical software stack you will see on any self-hoster / homelabber servers.

I am less inclined to mess with it now, as I don’t want to spend any more hours troubleshooting failing ingress, storage issues, pods not starting, or something else. All those layers involved in serving a request add up to the cognitive overload when something inevitably breaks.

However, I do realise I still crave experimentation. I want to try native routing with Cilium. I want a networking lab to study different routing setups. I want to try Ceph for storage, OVN for seamless migration of VMs. So many things.

Having homelab is how I become more confident at my job. However, I realise now that my setup is 80% infrastructure and supporting services, and 20% actual user-facing services. This is expected when running multi-node failover cluster with storage, networking layer, ingress, etc. However, it also consumes a lot of power.

Lenovo thinkcentre’s with debian13, incus and Kubernetes control plane node and a worker node take somewhere between 25-35 watts as a baseline. At .24 cents / kwh in the Netherlands this comes to ~6 EUR a month. Three of those will be close to 20 EUR. Ok for running experiments, not ok for 24/7 operation.

I know that my setup is way overkill. I did not care back then. Doing overkill and pushing boundaries of sensible is how I learned all of this. Now I want to have a clear separation of low-power, non-HA “production” services and my actual homelab. Perhaps, this transition is natural for many experienced homelabbers?

I started assessing what the bare minimum setup could be. There are still things I don’t want to compromise on.

Criteria the new production environment should meet

Automation

It needs to be largely self-driven and I need to have cookbooks for typical tasks. Machine goes down? Spinning up a new one should be a matter of installing an OS and executing runbooks (executed via a tool that rhymes with Gnarlsible).

It needs to be secure

Patching should mostly be autonomous, or extremely easy. Isolation should be on per-application / group of applications layer. Networking should be fully sealed and access permitted via controlled ingress instance.

It needs minimum infra components to apps ratio

This is the metric that I’d like to optimise for. Energy costs are high. Even though those m920q’s are not power hungry and can do everything I throw at it, I’d like to minimise CPU cycles to power my production services. I’d like to be able to switch off most of my lab when I am done for the day and have my production running with the smallest possible footprint.

It needs monitoring and alerting

My current setup monitoring is pretty basic. I have a blackbox exporter probing my private and public services endpoints and messaging me on Telegram when any of them are down for some time. In addition, node-exporter stock alerts (come with kube-prometheus-stack) alert on failed systemd services and other operational issues of the nodes.

The biggest loss here is high availability. When starting out, I did not properly assess this part. I did not question whether I need HA. I must had it. However, now I realise the cost of power it takes is not worth it for me. At the very worst case I will lose my production services during my travel and won’t be able to restore it for some time. And that’s ok. I do rely on my services day to day, but availability is not critical. I will get back to it when I can.

Split homelab and production setup

I’ve done a bit of poking around the web - places like r/selfhosted, r/homelab, hackernews. Seems that a single VM or bare-metal node with debian13 and docker is a solid start.

Managing compose files is usually done via some component with a stack concept. Each compose stack is managed as a set of applications together. Komodo caught my eye - it got a lot of praise. From my testing, it’s a very clean, no-nonsense stacks manager with auto-update, sync to/from git, and multi-node setup.

It also comes with docker swarm support. I am not sure it’s something I need as it’s leaning dangerously close to Kubernetes territory. I may need an additional node at some point, but for now, single node setup should do.

I also read a lot of great things about FreeBSD as a hypervisor. I scanned through their handbook and left impressed with the thorough setup guide on all aspects of the OS - installation, storage, networking, VMs, jails, etc.

My current Kubernetes setup should remain my primary homelab environment. There’s a lot of knowledge and automation created that should serve as a solid foundration to experiment with new networking stack, storage, seamless migration, perhaps even Kubevirt (although, I prefer Kubernetes in a VM not the other way around).

What I’ve done so far

I spent a few week-ends hand-crafting ansible setup for my FreeBSD hosts. One for ingress and one for hosting debian13 VM. Caddy as my reverse proxy I get certificates out of the box, easy installation via FreeBSD ports and running Caddy in jails - one for public services and one for private. Both isolated in their respective VLANs.

Ingress is only allowed to connect to the VM host specific ports where I run another Caddy instance that then forwards traffic to the services exposed on VMs internal IP.

I’ve been running it for a few weeks and it’s been solid. I was able to shut down 2 out of 4 hypervisor nodes. Most used services are now running in Docker and some still on Kubernetes which I am slowly migrating away.

If I mess up some deployment, reasoning about and troubleshooting my stack is way easier now so there has been a tangible benefit already.

But there are downsides, too. Kubernetes, albeit complex, solves many issues all homelabbers face in their journey. Service discovery. Kubernetes does it really, really well. All services can connect internally via predictable DNS entries. Where I had mysvc.ns now I have 10.30.0.20:9292. Having proper ingress makes it easier as now connecting via ports is only done in a single place, but I find it awkward still. I need to look into connecting Caddy or Traefik to services via Docker labels, but for now – adding a services via the port it’s exposed on is doing the job well enough.

I only need to figure out backups. Luckily, in FreeBSD running on ZFS I only have to do zfs send to my NAS and this should already be good enough and is super simple, seeing my NAS is already on ZFS.

I am stoked to have the separation in place because now I am not worried of breaking my production environment every time I want to mess with some settings and try things out.