Home Lab

Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

A little less than a year ago I wrote:

Everything is in shambles.

One machine is dead and probably not coming back. Services are randomly scattered amongst the survivors. My Kubernetes project is kinda-sorta paused.

This post is an addendum to that one, where I'm going to give an overview of what I've settled on, as much as one can settle in a homelab.

To recap a bit, in the previous post I had a bunch of machines that were mashed together into something resembling a coherent cluster:

  • hypnotoad, crushinator, and a precarious control plane VM running on a thin client
  • lrrr, an N150 box running Proxmox
  • nibbler, a piece of shit

Of those four machines, lrrr is the sole survivor.

The Problem of Nibbler

Nibbler, on television, is an adorable little three-eyed dude in Futurama that (courtesy spoiler warning, but 20 years is well past my cutoff) turns out to be a member of a hyperintelligent race of beings that can time travel and who, in fact, is the reason why Fry fell into the cryopod to start with.

Lord Nibbler, an adorable three-eyed alien from Futurama

nibbler, in my basement, was a machine that I had pinned a lot of hope on. It was a Lenovo M80s Gen3 SFF with an i7-12700 and DDR5 memory, picked up on eBay for what seemed like a steal (foreshadowing). I upgraded the memory to 128GB and the storage to 3TB of NVMe with left over pocket change. I put an HBA in it and hung a disk shelf off of it with seven 16TB drives. It was to be the core server, the one where all the action would happen.

my homelab stack circa February 2024. A big arrow is pointing at nibbler, the piece of shit lenovo

It was, as covered earlier, a piece of shit.

I'm still not quite sure what was wrong, but it was never reliable. Often it would just not come up after a reboot. Once in a while it would just forget about hardware.

On more than one occasion I or my spouse would discover that Jellyfin wouldn't play anything anymore, and I would shuffle down to the basement to attach a monitor and keyboard and discover that the entire storage disk shelf was errored out with inscrutable SAS adapter driver errors in the kernel logs.

Around the time I wrote the previous piece I put together a likely story. The eBay seller knew this machine was bad but they slapped an "eBay Certified Refurbished" label on anyway and sold it for a song to a sucker. By the time I realized this I had modified the machine so extensively I couldn't make a refund claim.

The Problem of Kubernetes

Simultaneous to the nibbler revelations I had another realization: Kubernetes is Too Hard. I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble. When the machine that has more cores and memory than every other machine in the cluster combined decides to self-immolate once or twice a week it puts a huge damper on the whole experiment.

My stop-gap solution was to revert everything important back to the way things were before k8s, which was effectively just docker compose. After letting that bake for a while I decided to just roll with it.

You Know What They Say About Temporary Solutions

It's now July of 2026.

lrrr exists in much the same form as it did before. It's still running Proxmox, but the only virtual things there are one LXC running Omada and one tiny VM that I've been using to ease myself into the agentic lifestyle. I've also done the worst thing imaginable to the Proxmox forum trolls: I've installed Docker on the host to run core services like Home Assistant and friends.

hypnotoad and crushinator are turned off, lying in state until I need them again.

morbo is a new-to-me HP Elitedesk 800 G3 SFF that runs TrueNAS SCALE and whose entire job is to manage storage and stay out of the way and it does a great job. The end.

ord-router is a VM running at a datacenter in Chicago that happens to have direct peering with Comcast. This is the public ingress node.

nibbler is now an amalgamation of old and new parts: same memory, same storage, new gamer-style case and motherboard and processor (Ryzen 9 9900X). It also recently gained an RTX 5060ti 16GB video card for local LLM experiments. This is the "everything else" machine. Everything that isn't critical to the house functioning as a house and the network as a network runs here.

Among other things, nibbler is where Forgejo, Minecraft, and Jellyfin and friends live, along with a bunch of other random crap.

There are also a handful of Raspberry Pi-class devices scattered around that are also deployed with this system, but they do appliance-type things: house-pi runs a Z-Wave stick and shed-pi is where the ADS-B receivers are plugged in.

Docker Compose with Extra Steps

Quirky? Yeah, let's go with quirky.

The software stack that deploys the homelab is arranged like this:

  • a bunch of spicy docker compose stacks
  • a handful of Ruby libraries that implement the spice
  • a Rakefile to drive the libraries
  • a YAML file that ties it all together

What's the spice, you ask?

  • cron jobs
  • secrets management
  • secure defaults
  • http ingress via Caddy with optional authn/authz and optional public-facing IP
  • automatic certificate management via Lego
  • backups
  • monitoring
  • dashboard
  • static websites served by the ingress
  • cross-stack dependency declarations
  • cross-platform docker image builds

Each compose stack declares what it wants using compose extensions (i.e. fields that start with x-). Here's a simple example:

services:
  metube:
    image: ghcr.io/alexta69/metube
    container_name: metube
    hostname: metube
    restart: unless-stopped
    x-security:
      enabled: true
    volumes:
      - media-nfs:/media
    environment:
      DOWNLOAD_DIR: /media/youtube
      HOST: '::'
    x-backup:
      enabled: false
    x-web:
      port: 8081
      auth: true
      dashboard:
        name: MeTube
        subtitle: Download YouTube Videos
        category: Media and Games

x-depends:
  - media-volume

This sets up MeTube, a little app that grabs videos from various places around the internet. It enables the secure defaults, it declares that it doesn't need backups, and then it sets up ingress. Here, x-web specifies the port, that it wants authentication/authorization, and that it should show up on the dashboard.

This example also illustrates the cross-stack dependencies. It pulls in the media-volume stack which sets up the media-nfs volume.

The trick that lets all that work is that all of the stacks get smushed into one big docker-compose file at deploy time. That means I don't have to do gross things like declare external volumes or networks, it all just hangs together.

Networking Is Type Two Fun

In a normal docker compose world every container in a stack shares a bridge network. For my purposes that doesn't make sense. I don't want some random vibe-coded thing that I downloaded yesterday to be able to talk directly to lldap, for example.

Instead, I emulate what you'd get if you were deploying individual stacks: each stack gets a bridge network to itself, and containers in other stacks can join it when necessary.

The natural way to implement ingress, then, would be to have every container that serves a website join a shared ingress bridge. Again, though, that doesn't really fit what I'm going for. I want to preserve the separation between stacks so that an extruded todo list manager that gets popped doesn't threaten everything else.

My solution is deterministic IPv6:

  • The fleet as a whole gets a /48 IPv6 ULA, which is similar in concept to an RFC1918 IPv4 network (ex: 10.10.10.0/24) but drawn from a vastly larger pool
  • Each host gets a /64 by hashing the hostname + salt for the next 16 bits
  • Each stack gets a /96 by hashing the stack name + salt for the next 32 bits
  • Each service gets a /128 by hashing the container name + salt for the last 32 bits

In other words:

<48 bit ULA prefix>:<16 bit host>:<32 bit stack>:<32 bit service>

Caddy runs with network_mode: host and all of the generated upstreams point at the IPv6 assigned to the containers, which works because the host naturally has access to all of the bridges.

Another weird/brave/stupid thing about my setup is that lrrr and nibbler both have ports 80 and 443 exposed to the internet over IPv6. Caddy is set up to reject non-local traffic unless the service specifically opts in with a public: true route. This is mostly for convenience so I don't have to set up Tailscale on mobile devices to access things like Jellyfin and the dashboard.

Oh, and Tailscale. Tailscale is so cool. Except on my iPhone which it makes very hot and drains the battery.

Tailscale is the backhaul for a few important things. Web traffic that hits ord-router is routed to the hosts in the basement over Tailscale. It's also how Caddy on each host talks to the central authn/authz system running on lrrr.

What Have We Learned Today

First, that I have not built a Kubernetes. Nothing about this changes at runtime so I don't need etcd or Raft or consensus of any sort. Everything is statically determined up front, with linters and tests and other nice things.

Second, the failure domains concept actually works really well in practice. lrrr is home for critical stuff which is kept to a minimum. nibbler is everything else that isn't a weird one-off appliance.

If nibbler goes down people are sad but not mad. If lrrr goes down the HVAC system gets a little dumber and the LEDs we rely on to tell us the cats are outside stop working.

Finally, it's really nice to have full control over the system. For example, the other day I decided to implement that static sites via ingress thing. I didn't have to dig for a solution on some kubernetes discourse that is five years out of date and then translate it forward. I just wrote some code and made it happen. Same with the IPv6 thing. Being able to do that smoothly is a consequence of having complete control.

Oh, and also, if none of this sounds like type one fun for you then sticking with Kubernetes or one of the zillion docker compose managers that people have put together is completely valid. I see you. It's ok.

I have a sanitized public snapshot of the code that builds the whole thing on my Forgejo if you're interested.

Wanna chat about this kind of thing? Links to my socials etc are in the footer.

It's 2026 so I guess this needs to be explicit: these are artisanal, human-produced words. The machines suggested fixes for my hackneyed clichés but I didn't use any of their words.

Homelab Failure Domains

Everything is in shambles.

One machine is dead and probably not coming back. Services are randomly scattered amongst the survivors. My Kubernetes project is kinda-sorta paused.

I'm unnerved and want to get it all sorted out, but I need to do some thinking out loud first.

My infrastructure currently consists of:

  • Omicron, a Kubernetes cluster that is just barely functioning. It is currently running this site, VMSave, and basically nothing else. Consists of hypnotoad, crushinator, and a control plane VM running on a Wyse 5070 running Proxmox.

  • Nimbus, a Kubernetes cluster that is not functioning at all. I tried building out a GitOps-driven cluster as my second attempt and everything was going swimmingly until nibbler, a historically unreliable piece of hardware that when it works has more cores and memory that the rest of my infra combined, fell over again in yet another inexplicable way.

  • Lrrr, a box with an Intel N150 and 32GB of memory running a VM on top of Proxmox that is hosting almost everything that was previously on the two Kubernetes clusters.

This jumbled state of affairs is basically due to a series of impulsive hardware purchases and "oh that's neat, let's do that" infrastructure changes.

Let's talk about failure domains.

I think of a failure domain as a set of risks and mitigation strategies as applied to a particular instance of a service.

The canonical example in the software-as-a-service world is "production", i.e. the instance of the service that the customers touch. The one that makes the money. The primary risk is the money going away if the service goes down.

A SaaS shop may have a staging environment, where changes get tested before they hit production. The main risk in staging is inconveniencing your coworkers, but the consequences of that to the company are much less impactful.

Each developer in then hopefully has one or more of their own environments in which to actually make the software. These are practically risk free to the company as a whole, only inconveniencing one developer if something goes awry.

Overcomplicated home infrastructure doesn't map neatly into the same failure domains as a SaaS business, of course, but they still exist.

When I think about the users of the services in my home I imagine a sort of abstract "household delight" score. Points accrue implicitly when things are running fine and people are able to use the things I'm trying to provide. Points get deducted when they notice things aren't working or when they see me stomping around grumbling about full hard drives and boot errors.

By that logic I have three different failure domains (actually four but we'll get to that):

  1. Critical production. The absence of service would be immediately noticed and commented upon, often affecting the comfort of the occupants of the house. Examples: network, DNS, Home Assistant and friends, IoT coordinators.

  2. Production: The absence of service would be noticed eventually but even an extended outage wouldn't cause hardship. Examples: Jellyfin, Sonarr and friends, paperless-ngx.

  3. Lab: I'm the only one affected by things breaking in the lab. A playground for testing and fucking around.

The fourth failure domain that doesn't neatly map into the above is production services for external users. VMSave and this site are the big ones but there are a few smaller things too.

When I'm brutally honest with myself I have to recognize that the biggest common source of failure in every domain is me. Trying things, adding hardware, replacing software, messing around, testing in production.

Often my partner will remark "I don't understand how things just fail!" They usually don't. Failure is an immediate or delayed result of me changing something without considering the impact.

So. What to do.

Obviously first I need to delineate the lab from everything else. Separate hardware for sure, maybe even hide it all behind another router and subnet.

For production, one plan would be to just put everything critical and production on the one docker VM and let it be. The machine isn't struggling overall but Jellyfin isn't super great because the N150 doesn't quite have the oomph necessary to transcode some of the stuff we have in real time.

Another plan would be to split them onto two machines running docker VMs. This would reduce the churn on critical production and reduce the chances of a change messing things up.

Yet another plan would be to spin up a separate Kubernetes cluster for each, moving right along the overcomplicated continuum.

The thing is, Kubernetes makes sense to me now that I've worked with it in anger a little. I really think for my application it makes sense, and the problems with Nimbus come down to nibbler being flaky and k8s trying to self-heal without enough resources available.

I don't know what to do about external production. My intent was to have it at home out of principle (or maybe out of spite) but it would probably be better to have it in an isolated cloud environment.

The one Docker VM is working ok, but it's mixing failure domains which makes me uncomfortable. For now, things are how they are and I can't let myself worry about it too much.

Links in the footer if you have comments or ideas. I'd love to hear them.

Kubernetes at the (homelab) Edge

As I've mentioned before, the RF environment in my house is difficult. The house layout is roughly:

  • single level house, half of which has a basement under it
  • single level office / mother-in-law-suite / ADU / whatever you want it
  • two car garage in between
  • shed in the back yard

The house and ADU are built out of cinder block and brick on the outside and plasterboard (not drywall) on the inside with foil-backed insulation sandwiched in between. The garage is built in between the two buildings, half with the same cinder block and brick construction and half with more modern stick construction.

Diagram of my backyard fiber project

These buildings were built in the 1950s when labor was cheap, longevity was valued, and AM radio stations were extremely powerful.

The shed is just an ordinary stick and OSB sheathing shed, but it's quite a distance from the house proper.

I consider the three buildings and the garage as separate "RF zones", for lack of a better word. Zigbee and Wi-Fi at 2.4GHz and Wi-Fi 5 at 5GHz do not propagate through the block and foil very well or at all. Z-Wave (900MHz) has a slightly better time but because the house is so spread out and Z-Wave has a fairly low hop limit for mesh packets (4 hops, vs at least 15 for Zigbee) repeaters don't work very well. Lutron Caseta (434MHz) has phenomenal range and penetrates the foil with zero issues, but the device variety is severely limited. In particular, there are no Caseta smart locks.

Each zone has:

  • At least one Wi-Fi access point
  • A Z-Wave hub
  • (sometimes) a Zigbee hub
  • (sometimes) RS232 or RS485 to USB converters for equipment like our generator and furnace

Previous Solutions

The Z-Wave and Zigbee "hubs" are mostly just USB sticks stuck into a free port on a Dell Wyse 3040 thin client. I've had these little machines running for almost five years through a few different setups.

a Dell Wyse 3040 hanging on a wall with a PoE splitter hanging next to it

First, I had ser2net running over Tailscale and had the gateway software, i.e. zwave-js-ui and zigbee2mqtt, running on a server in my house. This was fine, until I had some significant clock skew when a node rebooted (i.e. the CMOS battery was dead and the hunk of junk thought it was 2016) and Tailscale refused to start because it thought the SSL certificate for the Tailscale control plane wasn't valid.

The second draft was to just run the gateway software directly on the 3040s. This actually works fine. The 3040s are capable machines, roughly comparable to a Raspberry Pi 4, so they could run a little javascript program just fine. It was somewhat less responsive than running the gateway on the server, though.

The third version of this is to use a hardware gateway. I'm currently using one for Zigbee because the location of the house zone's 3040 is not ideal for some important Zigbee devices and they lose connection a lot. I positioned the hardware gateway in a spot that has good Wi-Fi coverage but no Ethernet port and now those devices are rock solid.

But what if Kubernetes?

While rolling out Kubernetes I didn't really plan on converting the 3040s because running the gateway software on them was working fine. Just as an experiment I attempted to install Talos on one of my spares. Amazingly, it worked great after making one tweak to the install image. The 3040s are very particular about certain things and they, like many of the SBCs that Talos supports, don't like the swanky console dashboard. After turning that off the machine came right up as a Kubernetes node in the cluster.

At idle the Kubernetes workloads plus my cluster's standard DaemonSet pods use about 40% of the machine's 2GiB of memory and roughly 30% of CPU.

talosctl dashboard viewing one of my Dell Wyse 3040s

That leaves way more than enough to run ser2net.

Automatic ser2net Config

I initially thought that I would use Akri to spawn ser2net. Akri is a project that came out of Microsoft that acts as a generic device plugin for Kubernetes as well as managing what they call "brokers", which are just programs that attach to whatever device and provide it to the cluster.

That sounded perfect for my purposes so I set it up and let it bake for a few days. It did not go well.

The big problem is that Akri is just not very stable. Things were randomly falling over in such a way that the Akri-managed Z-Wave ser2net brokers would crash loop overnight. I made no debugging progress so I started on my fallback idea: automatically managing a ser2net config.

Realistically, my needs are simple:

  • I want to present one or more serial devices to the network.
  • I want this to be reliable.
  • I want this to be secure.
  • I don't want to micromanage it.

It turns out, discovering USB serial devices on linux is actually pretty trivial. You just have to follow a bunch of symlinks.

This shell script is based on logic found in the go-serial project:

set -e
set -x

# Find all of the TTYs with names we might be interested in
ttys=$(ls /sys/class/tty | egrep '(ttyS|ttyHS|ttyUSB|ttyACM|ttyAMA|rfcomm|ttyO|ttymxc)[0-9]{1,3}')

for tty in $ttys; do
    # follow the symlink to find the real device
    realDevice=$(readlink -f /sys/class/tty/$tty/device)

    # what subsystem is it?
    subsystem=$(basename $(readlink -f $realDevice/subsystem))

    # locate the directory where the usb information is
    usbdir=""
    if [ "$subsystem" = "usb-serial" ]; then
        # usb-serial is two levels up from the tty
        usbdir=$(dirname $(dirname $realDevice))
    elif [ "$subsystem" = "usb" ]; then
        # regular usb is one level up from the tty
        usbdir=$(dirname $realDevice)
    else
        # we don't care about this device
        continue
    fi

    # read the productId and vendorId attributes from the USB device
    productId=$(cat $usbdir/idProduct)
    vendorId=$(cat $usbdir/idVendor)

    snippetFile="$vendorId:$productId.yaml"

    if [ -f "$snippetFile" ]; then
        sed "s/DEVNODE/\/dev\/$tty/" $snippetFile
    fi
done

The last few lines of the loop body look for a YAML snippet available for the specific vendorId/productId pair. If there is one, replace the constant DEVNODE with the actual device path and write it to stdout.

Here's what one of the snippets looks like:

# zigbee

connection: &zigbee
  accepter: tcp,6639
  connector: serialdev,DEVNODE,115200n81,local,dtr=off,rts=off
  options:
    kickolduser: true

This snippet tells ser2net to open a listening connection on TCP port 6639 and wire it to a serial device at path DEVNODE. Use 115200n81 parity and keep the dtr and rts bits off (this is specific to the Sonoff zigbee stick I'm using). Further, when a new connection opens immediately kick off the old one.

I drop the above script, the YAML snippets, and this simple entrypoint.sh script into a container image based on the standard ser2net container.

#!/bin/bash

set -e
set -x
set -o pipefail

echo "Generating ser2net.yaml"

mkdir -p /etc/ser2net
./discover.sh > /etc/ser2net/ser2net.yaml

echo "Running ser2net"

cat /etc/ser2net/ser2net.yaml

exec ser2net -d -l -c /etc/ser2net/ser2net.yaml

I then deploy it to my cluster as a DaemonSet targeting nodes labeled with keen.land/serials=true:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ser2net
  namespace: iot-system
  labels:
    app: ser2net
spec:
  selector:
    matchLabels:
      app: ser2net
  template:
    metadata:
      labels:
        app: ser2net
    spec:
      nodeSelector:
        keen.land/serials: "true"
      volumes:
        - name: devices
          hostPath:
            path: /dev
      imagePullSecrets:
        - name: ghcr.io
      containers:
        - name: ser2net
          image: "ghcr.io/peterkeen/ser2net-auto:main"
          securityContext:
            privileged: true
          volumeMounts:
            - name: devices
              mountPath: /dev
      restartPolicy: Always
      hostNetwork: true

The interesting things here are that the pod is running in privileged mode on the host network. I thought about using container ports but then I would have to somehow know what DaemonSet pod name mapped to which host. With hostNetwork: true I don't have to think about that and can just use the host's name in my gateway configs. There's an opportunity here to cook up a custom deployment type with something like Metacontroller, which I have installed in the cluster but as of yet haven't done anything with.

Pros and Cons

So all of this work and what do I have? Basically what I had when I started:

  • ser2net on the devices with the serial ports
  • gateway software running on a server

The big pro of this setup is that I have one consistent management interface. I can set up and tear down ser2net with the exact same interface I use to set up and tear down everything else in the cluster.

There are a couple of cons, too. First, this probably uses a touch more power than the old solution because in addition to ser2net the Wyse 3040s are running all of the Kubernetes infrastructure. These things use so little power as is that I don't think it really matters, but it's worth pointing out.

Second, there's more to go wrong. Before this I had Alpine running basically nothing except ser2net. The system was static in practice, meaning that there was very little that could break.

Now there are several components running all the time that could break with a bad upgrade and could require me to take a crash cart out to each machine.

This is also putting more stress on the drives. All of these machines are booting off of significantly overprovisioned high endurance SD cards now so that shouldn't be an issue, but it's still something to keep in mind. The nice thing is that they're entirely stateless so swapping the card and reinstalling should be a quick operation.

Ultimately I think this is a good move and I plan to continue down the path of making every non-laptop device run on Kubernetes with very few exceptions.

Switching to Kubernetes

And you may ask yourself, "How do I work this?"
And you may ask yourself, "Where is that large home server?"

Once upon a time I had a Mac mini. It was hooked up to the tv (because we only had the one) and it ran Plex. It was fine.

Later, my new spouse and I moved across the country into a house. I decided that I should get a server because I was going to be a big time consultant and I figured I would need a staging environment. A Dell T30 picked up on super sale arrived soon after.

The server sat, ignored, while we suffered through the first few years of one baby, then two babies.

Later, we moved to our forever house and I found Home Assistant. I picked up a Raspberry Pi 4. All was good.

Except it kind of sucked? A 1GB Pi 4 is pretty limited in what it can practically run. Home Assistant ran mostly ok but anything else was beyond it's capabilities. To eBay!

Oooh, shiny hardware

Over the past four years I've accumulated a modest menagerie of hardware:

  • Hypnotoad, a HP 800 G3 mini
  • Crushinator, a HP 800 G3 SFF
  • Morbo, another HP 800 G3 SFF
  • Lrrr, a Dell Wyse 5070 thin client
  • Roberto, another Dell Wyse 5070
  • Nibbler, a Lenovo M80S Gen 3 SFF
  • Shed, another Dell Wyse 5070 (such a boring name)
  • A pack of roving Dell Wyse 3040 thin clients
  • The original Pi 4

The T30, sadly, imploded when I tried to install a video card and fried the motherboard. Its name was Kodos and it was a good box.

Software, take 1 through N

As I was acquiring hardware I was also acquiring software to run on it and developed a somewhat esoteric way of deploying that software. The first interesting version was a self-deploying Docker container. It would get passed the Docker socket and run compose, deciding on the fly what to deploy based on the hostname of the machine.

This was fine, but it proved too much for the 3040s which have fragile 8GB eMMC drives.

A later version moved the script to my laptop and used Ansible to push Docker compose files out to all the machines.

Fine. Fiddly, but fine.

Software, take N + 1

Xe Iaso is a person that I've been following online for years. Recently they went through a homelab transformation, where for Reasons they decided to switch away from NixOS. After trying various things, much to everyone's chagrin, they settled on Kubernetes running on Talos.

Talos seemed to be what I wanted: an immutable, hardened OS designed for one thing and one thing only: Kubernetes.

Much like Xe, I had resisted Kubernetes at home for a long time. Too complex. Too much overhead. Just too much.

Taking another look at that hardware list, though, I do actually have a somewhat Kubernetes-shaped problem. I want to treat my hardware as respected but mostly interchangeable pets.

My deployment script was sophisticated but had no ability to just put something somewhere else automatically. It was entirely static, so when something needed to move I would have to restore a backup to the new target and manually redeploy at least part of the world in order to get the ingress set up properly.

Kubernetes takes care of that stuff for me. I don't have to think about where any random workload runs and I don't have to think about migrating it somewhere else if the node falls over. DNS, SSL certificates, backups, it all just happens in the background.

What's it look like?

After a couple of weeks of futzing around and day dreaming I settled on this software stack:

The next thing to decide was how to divide up the hardware into control plane and worker nodes. Here's what I have so far:

  • Three (3) control plane nodes: hypnotoad, crushinator, and lrrr
  • Seven (7) local worker nodes: hypnotoad, crushinator, nibbler, shed, three Wyse 3040s hosting Z-Wave sticks
  • One (1) cloud worker node

You might notice that several nodes are doing double duty.

Splitting the control plane off to dedicated nodes makes sense when you have a fleet of hundreds of machines in a data center. I don't have that.

A small VM running on Lrrr is the only dedicated control plane node. The only reason for that is because Lrrr also hosts my Unifi and Omada network controllers and I haven't worked up the gumption to move those from Proxmox LXCs to k8s workloads.

Hypnotoad, Crushinator, and Nibbler are general compute. Nibbler has an Nvidia Tesla P4 GPU, which is not particularly impressive but fun to play with. Both Hypnotoad and Nibbler have iGPUs capable of running many simultaneous Jellyfin streams. Crushinator is a VM taking up most of the host which is also serving as a backup NAS for non-media data.

Shed lives in the shed and is connected to a bunch of USB devices, including two SDR radios, a Z-Wave stick, and an RS232-to-USB adapter for the generator.

Morbo is running TrueNAS and has no connection to Kubernetes except that some stuff running in k8s uses NFS shares. It's also the backup target for Longhorn and the script I use to backup Talos' etcd database.

Self-hosting in the Cloud

Talos has a neat feature built in that they call KubeSpan. This is a Wireguard mesh network between all of the nodes in the cluster that uses a hosted discovery service to exchange public keys.

Essentially, you can flip a single option in your Talos configs and have all of your nodes meshed, with a bonus option to send all internal cluster traffic over the Wireguard interface. The discovery service never sees private data, just hashes. It's really cool.

I'm using KubeSpan to put one of my nodes on a VPS to get a public IP without exposing my home ISP connection directly. After initial setup I was able to change the firewall to block all inbound ports other than 80, 443, and the KubeSpan UDP port.

To actually serve public traffic I installed a separate instance of ingress-nginx that only runs on the cloud node. This instance is set up to directly expose the cloud node's public IP which gets picked up by external-dns automatically.

I'm still trying to decide if this single node is enough or if I should get really clever and use a proxy running on Fly to get a public anycast IP.

Ok, but what's running?

Learning how Kubernetes works has been great and this process filled in quite a few gaps in my understanding, but it probably wouldn't have been worth the effort without hosting something useful.

Currently I'm hosting a couple of external production workloads:

  • this blog
  • VMSave
  • a handful of very small websites

I'm also running a bunch of homeprod services:

  • Home Assistant
  • Whisper and Piper, speech-to-text and text-to-speech tools and components of the Home Assistant voice pipeline
  • four (4) instances of Z-Wave JS UI, one per RF "zone" (this house has wacky RF behavior)
  • two (2) instances of Zigbee2MQTT, one in each RF zone that has Zigbee devices
  • Genmon keeps tabs on our whole home standby generator
  • A Minecraft server for me and my kids
  • Paperless-ngx stores and indexes important documents
  • Ultrafeeder puts the planes flying overhead on a map
  • iCloudPD-web for effortless iCloud photo backups
  • Jellyfin, an indexer and server for TV shows, movies and music
  • Sonarr, Radarr, Prowlarr and SABnzbd form the core of our media acquisition system
  • Jellyseerr makes requesting new media easy for the other people who live in the house
  • Calibre Web Automated is an amazing tool that serves all of my eBooks to my Kobo eReader
  • Ollama and Open Web UI for dinking around with local LLMs
  • Homer to keep track of all of the above, set as my browser homepage

Left To Do

There are a few things left on the todo list. Roberto is hooked up to a webcam that watches my 3D printer and I haven't touched it yet because it is connected via Wi-Fi which Talos doesn't support at all.

I also haven't touched the raspberry pi, mostly for the same reason. The pi is serving as a gateway between a Wi-Fi SD card that lives in my CPAP machine and the rest of the network so that I can scrape the data off without having to pull the card or futz with my laptop's Wi-Fi every day.

The Wi-Fi SD card, you see, only exists as an access point. It cannot be put into a mode that connects to another network. The pi has a USB Wi-Fi adapter connected to the card's network and the built-in Wi-Fi connected to the home network with nginx in between serving as a proxy. I don't think this is something that I really want or need to move into k8s.

I want to set up some sort of central authn/authz system for the homeprod services. The current fashion seems to be Pocket ID but I haven't been able to get it working reliably.

I'm thinking about setting up a small ActivityPub server to play around with.

A photo viewer like Immich might be cool to set up.

Overkill?

Of course this is overkill. This could probably all live on a single Wyse 5070 with a couple big harddrives attached.

I think it's been worth it to use Kubernetes in anger. I'm really enjoying the ability to deploy whatever I want to the cluster without having to think about where it runs, where it stores data, etc.

I've also learned a ton and fixed a bunch of preconceived notions and it's already helped increase reliability in a few things that affect household acceptance in big ways.

Homeprod Management with Docker Compose

Recently I decided to change how I manage my homeprod environment (home production, i.e. the things that other people in my household rely on and tell me if they're down). I moved everything over to docker-compose stacks managed with a small(ish) shell script. Skip down a bit if you don't want backstory.

Z-Wave Controllers using Dell Wyse Thin Clients

About two years ago my family and I moved into our (hopefully) forever home. This house has an unfortunately large number of exterior doors and is situated on a main road, so we wanted to make sure that most of the doors are locked without having to patrol the doors before bedtime every night.

To that end, I put z-wave locks on every door and built out a z-wave network so they could be reached from my Home Assistant instance that shows their status on a nice tidy dashboard.