One machine is dead and probably not coming back. Services are randomly scattered amongst the survivors. My Kubernetes project is kinda-sorta paused.

I'm unnerved and want to get it all sorted out, but I need to do some thinking out loud first.

My infrastructure currently consists of:

• Omicron, a Kubernetes cluster that is just barely functioning. It is currently running this site, VMSave, and basically nothing else. Consists of hypnotoad, crushinator, and a control plane VM running on a Wyse 5070 running Proxmox.

• Nimbus, a Kubernetes cluster that is not functioning at all. I tried building out a GitOps-driven cluster as my second attempt and everything was going swimmingly until nibbler, a historically unreliable piece of hardware that when it works has more cores and memory that the rest of my infra combined, fell over again in yet another inexplicable way.

• Lrrr, a box with an Intel N150 and 32GB of memory running a VM on top of Proxmox that is hosting almost everything that was previously on the two Kubernetes clusters.

This jumbled state of affairs is basically due to a series of impulsive hardware purchases and "oh that's neat, let's do that" infrastructure changes.

Let's talk about failure domains.

I think of a failure domain as a set of risks and mitigation strategies as applied to a particular instance of a service.

The canonical example in the software-as-a-service world is "production", i.e. the instance of the service that the customers touch. The one that makes the money. The primary risk is the money going away if the service goes down.

A SaaS shop may have a staging environment, where changes get tested before they hit production. The main risk in staging is inconveniencing your coworkers, but the consequences of that to the company are much less impactful.

Each developer in then hopefully has one or more of their own environments in which to actually make the software. These are practially risk free to the company as a whole, only inconvenicing one developer if something goes awry.

Overcomplicated home infrastructure doesn't map neatly into the same failure domains as a SaaS business, of course, but they still exist.

When I think about the users of the services in my home I imagine a sort of abstract "household delight" score. Points accrue implicitly when things are running fine and people are able to use the things I'm trying to provide. Points get deducted when they notice things aren't working or when they see me stomping around grumbling about full hard drives and boot errors.

By that logic I have three different failure domains (actually four but we'll get to that):

1. Critical production. The absense of service would be immediately noticed and commented upon, often affecting the comfort of the occupants of the house. Examples: network, DNS, Home Assistant and friends, IoT coordinators.

2. Production: The absense of service would be noticed eventually but even an extended outage wouldn't cause hardship. Examples: Jellyfin, Sonarr and friends, paperless-ngx.

3. Lab: I'm the only one affected by things breaking in the lab. A playground for testing and fucking around.

The fourth failure domain that doesn't neatly map into the above is production services for external users. VMSave and this site are the big ones but there are a few smaller things too.

When I'm brutally honest with myself I have to recognize that the biggest common source of failure in every domain is me. Trying things, adding hardware, replacing software, messing around, testing in production.

Often my partner will remark "I don't understand how things just fail!" They usually don't. Failure is an immediate or delayed result of me changing something without considering the impact.

So. What to do.

Obviously first I need to deliniate the lab from everything else. Separate hardware for sure, maybe even hide it all behind another router and subnet.

For production, one plan would be to just put everything critical and production on the one docker VM and let it be. The machine isn't struggling overall but Jellyfin isn't super great because the N150 doesn't quite have the oomph necessary to transcode some of the stuff we have in real time.

Another plan would be to split them onto two machines running docker VMs. This would reduce the churn on critical production and reduce the chances of a change messing things up.

Yet another plan would be to spin up a separate Kubernetes cluster for each, moving right along the overcomplicated continuum.

The thing is, Kubernetes makes sense to me now that I've worked with it in anger a little. I really think for my application it makes sense, and the problems with Nimbus come down to nibbler being flaky and k8s trying to self-heal without enough resources available.

I don't know what to do about external production. My intent was to have it at home out of principle (or maybe out of spite) but it would probably be better to have it in an isolated cloud environment.

The one Docker VM is working ok, but it's mixing failure domains which makes me uncomfortable. For now, things are how they are and I can't let myself worry about it too much.

Links in the footer if you have comments or ideas. I'd love to hear them.

• single level house, half of which has a basement under it
• single level office / mother-in-law-suite / ADU / whatever you want it
• two car garage in between
• shed in the back yard

The house and ADU are built out of cinder block and brick on the outside and plasterboard (not drywall) on the inside with foil-backed insulation sandwiched in between. The garage is built in between the two buildings, half with the same cinder block and brick construction and half with more modern stick construction.

These buildings were built in the 1950s when labor was cheap, longevity was valued, and AM radio stations were extremely powerful.

The shed is just an ordinary stick and OSB sheathing shed, but it's quite a distance from the house proper.

I consider the three buildings and the garage as separate "RF zones", for lack of a better word. Zigbee and Wi-Fi at 2.4GHz and Wi-Fi 5 at 5GHz do not propagate through the block and foil very well or at all. Z-Wave (900MHz) has a slightly better time but because the house is so spread out and Z-Wave has a fairly low hop limit for mesh packets (4 hops, vs at least 15 for Zigbee) repeaters don't work very well. Lutron Caseta (434MHz) has phenominal range and penetrates the foil with zero issues, but the device variety is severely limited. In particular, there are no Caseta smart locks.

Each zone has:

• At least one Wi-Fi access point
• A Z-Wave hub
• (sometimes) a Zigbee hub
• (sometimes) RS232 or RS485 to USB converters for equipment like our generator and furnace

Previous Solutions

The Z-Wave and Zigbee "hubs" are mostly just USB sticks stuck into a free port on a Dell Wyse 3040 thin client. I've had these little machines running for almost five years through a few different setups.

First, I had ser2net running over Tailscale and had the gateway software, i.e. zwave-js-ui and zigbee2mqtt, running on a server in my house. This was fine, until I had some significant clock skew when a node rebooted (i.e. the CMOS battery was dead and the hunk of junk thought it was 2016) and Tailscale refused to start because it thought the SSL certificate for the Tailscale control plane wasn't valid.

The second draft was to just run the gateway software directly on the 3040s. This actually works fine. The 3040s are capable machines, roughly comparable to a Raspberry Pi 4, so they could run a little javascript program just fine. It was somewhat less responsive than running the gateway on the server, though.

The third version of this is to use a hardware gateway. I'm currently using one for Zigbee because the location of the house zone's 3040 is not ideal for some important Zigbee devices and they lose connection a lot. I positioned the hardware gateway in a spot that has good Wi-Fi coverage but no Ethernet port and now those devices are rock solid.

But what if Kubernetes?

While rolling out Kubernetes I didn't really plan on converting the 3040s because running the gateway software on them was working fine. Just as an experiment I attempted to install Talos on one of my spares. Amazingly, it worked great after making one tweak to the install image. The 3040s are very particular about certain things and they, like many of the SBCs that Talos supports, don't like the swanky console dashboard. After turning that off the machine came right up as a Kubernetes node in the cluster.

At idle the Kubernetes workloads plus my cluster's standard DaemonSet pods use about 40% of the machine's 2GiB of memory and roughly 30% of CPU.

That leaves way more than enough to run ser2net.

Automatic ser2net Config

I initially thought that I would use Akri to spawn ser2net. Akri is a project that came out of Microsoft that acts as a generic device plugin for Kubernetes as well as managing what they call "brokers", which are just programs that attach to whatever device and provide it to the cluster.

That sounded perfect for my purposes so I set it up and let it bake for a few days. It did not go well.

The big problem is that Akri is just not very stable. Things were randomly falling over in such a way that the Akri-managed Z-Wave ser2net brokers would crash loop overnight. I made no debugging progress so I started on my fallback idea: automatically managing a ser2net config.

Realistically, my needs are simple:

• I want to present one or more serial devices to the network.
• I want this to be reliable.
• I want this to be secure.
• I don't want to micromanage it.

It turns out, discovering USB serial devices on linux is actually pretty trivial. You just have to follow a bunch of symlinks.

This shell script is based on logic found in the go-serial project:

set -e
        set -x
        
        # Find all of the TTYs with names we might be interested in
        ttys=$(ls /sys/class/tty | egrep '(ttyS|ttyHS|ttyUSB|ttyACM|ttyAMA|rfcomm|ttyO|ttymxc)[0-9]{1,3}')
        
        for tty in $ttys; do
        # follow the symlink to find the real device
        realDevice=$(readlink -f /sys/class/tty/$tty/device)
        
        # what subsystem is it?
        subsystem=$(basename $(readlink -f $realDevice/subsystem))
        
        # locate the directory where the usb information is
        usbdir=""
        if [ "$subsystem" = "usb-serial" ]; then
        # usb-serial is two levels up from the tty
        usbdir=$(dirname $(dirname $realDevice))
        elif [ "$subsystem" = "usb" ]; then
        # regular usb is one level up from the tty
        usbdir=$(dirname $realDevice)
        else
        # we don't care about this device
        continue
        fi
        
        # read the productId and vendorId attributes from the USB device
        productId=$(cat $usbdir/idProduct)
        vendorId=$(cat $usbdir/idVendor)
        
        snippetFile="$vendorId:$productId.yaml"
        
        if [ -f "$snippetFile" ]; then
        sed "s/DEVNODE/\/dev\/$tty/" $snippetFile
        fi
        done
        

The last few lines of the loop body look for a YAML snippet available for the specific vendorId/productId pair. If there is one, replace the constant DEVNODE with the actual device path and write it to stdout.

Here's what one of the snippets looks like:

# zigbee
        
        connection: &zigbee
        accepter: tcp,6639
        connector: serialdev,DEVNODE,115200n81,local,dtr=off,rts=off
        options:
        kickolduser: true
        

This snippet tells ser2net to open a listening connection on TCP port 6639 and wire it to a serial device at path DEVNODE. Use 115200n81 parity and keep the dtr and rts bits off (this is specific to the Sonoff zigbee stick I'm using). Further, when a new connection opens immediately kick off the old one.

I drop the above script, the YAML snippets, and this simple entrypoint.sh script into a container image based on the standard ser2net container.

#!/bin/bash
        
        set -e
        set -x
        set -o pipefail
        
        echo "Generating ser2net.yaml"
        
        mkdir -p /etc/ser2net
        ./discover.sh > /etc/ser2net/ser2net.yaml
        
        echo "Running ser2net"
        
        cat /etc/ser2net/ser2net.yaml
        
        exec ser2net -d -l -c /etc/ser2net/ser2net.yaml
        

I then deploy it to my cluster as a DaemonSet targeting nodes labeled with keen.land/serials=true:

apiVersion: apps/v1
        kind: DaemonSet
        metadata:
        name: ser2net
        namespace: iot-system
        labels:
        app: ser2net
        spec:
        selector:
        matchLabels:
        app: ser2net
        template:
        metadata:
        labels:
        app: ser2net
        spec:
        nodeSelector:
        keen.land/serials: "true"
        volumes:
        - name: devices
        hostPath:
        path: /dev
        imagePullSecrets:
        - name: ghcr.io
        containers:
        - name: ser2net
        image: "ghcr.io/peterkeen/ser2net-auto:main"
        securityContext:
        privileged: true
        volumeMounts:
        - name: devices
        mountPath: /dev
        restartPolicy: Always
        hostNetwork: true
        

The interesting things here are that the pod is running in privileged mode on the host network. I thought about using container ports but then I would have to somehow know what DaemonSet pod name mapped to which host. With hostNetwork: true I don't have to think about that and can just use the host's name in my gateway configs. There's an opportunity here to cook up a custom deployment type with something like Metacontroller, which I have installed in the cluster but as of yet haven't done anything with.

Pros and Cons

So all of this work and what do I have? Basically what I had when I started:

• ser2net on the devices with the serial ports
• gateway software running on a server

The big pro of this setup is that I have one consistent management interface. I can set up and tear down ser2net with the exact same interface I use to set up and tear down everything else in the cluster.

There are a couple of cons, too. First, this probably uses a touch more power than the old solution because in addition to ser2net the Wyse 3040s are running all of the Kubernetes infrastructure. These things use so little power as is that I don't think it really matters, but it's worth pointing out.

Second, there's more to go wrong. Before this I had Alpine running basically nothing except ser2net. The system was static in practice, meaning that there was very little that could break.

Now there are several components running all the time that could break with a bad upgrade and could require me to take a crash cart out to each machine.

This is also putting more stress on the drives. All of these machines are booting off of significantly overprovisioned high endurance SD cards now so that shouldn't be an issue, but it's still something to keep in mind. The nice thing is that they're entirely stateless so swapping the card and reinstalling should be a quick operation.

Ultimately I think this is a good move and I plan to continue down the path of making every non-laptop device run on Kubernetes with very few exceptions.

]]>

And you may ask yourself, "How do I work this?"
And you may ask yourself, "Where is that large home server?"

Once upon a time I had a Mac mini. It was hooked up to the tv (because we only had the one) and it ran Plex. It was fine.

Later, my new spouse and I moved across the country into a house. I decided that I should get a server because I was going to be a big time consultant and I figured I would need a staging environment. A Dell T30 picked up on super sale arrived soon after.

The server sat, ignored, while we suffered through the first few years of one baby, then two babies.

Later, we moved to our forever house and I found Home Assistant. I picked up a Raspberry Pi 4. All was good.

Except it kind of sucked? A 1GB Pi 4 is pretty limited in what it can practically run. Home Assistant ran mostly ok but anything else was beyond it's capabilities. To eBay!

Oooh, shiny hardware

Over the past four years I've accumulated a modest menagerie of hardware:

• Hypnotoad, a HP 800 G3 mini
• Crushinator, a HP 800 G3 SFF
• Morbo, another HP 800 G3 SFF
• Lrrr, a Dell Wyse 5070 thin client
• Roberto, another Dell Wyse 5070
• Nibbler, a Lenovo M80S Gen 3 SFF
• Shed, another Dell Wyse 5070 (such a boring name)
• A pack of roving Dell Wyse 3040 thin clients
• The original Pi 4

The T30, sadly, imploded when I tried to install a video card and fried the motherboard. It's name was Kodos and it was a good box.

Software, take 1 through N

As I was acquiring hardware I was also acquiring software to run on it and developed a somewhat esoteric way of deploying that software. The first interesting version was a self-deploying Docker container. It would get passed the Docker socket and run compose, deciding on the fly what to deploy based on the hostname of the machine.

This was fine, but it proved too much for the 3040s which have fragile 8GB eMMC drives.

A later version moved the script to my laptop and used Ansible to push Docker compose files out to all the machines.

Fine. Fiddly, but fine.

Software, take N + 1

Xe Iaso is a person that I've been following online for years. Recently they went through a homelab transformation, where for Reasons they decided to switch away from NixOS. After trying various things, much to everyone's chagrin, they settled on Kubernetes running on Talos.

Talos seemed to be what I wanted: an immutable, hardened OS designed for one thing and one thing only: Kubernetes.

Much like Xe, I had resisted Kubernetes at home for a long time. Too complex. Too much overhead. Just too much.

Taking another look at that hardware list, though, I do actually have a somewhat Kubernetes-shaped problem. I want to treat my hardware as respected but mostly interchangable pets.

My deployment script was sophisticated but had no ability to just put something somewhere else automatically. It was entirely static, so when something needed to move I would have to restore a backup to the new target and manually redeploy at least part of the world in order to get the ingress set up properly.

Kubernetes takes care of that stuff for me. I don't have to think about where any random workload runs and I don't have to think about migrating it somewhere else if the node falls over. DNS, SSL certificates, backups, it all just happens in the background.

What's it look like?

After a couple of weeks of futzing around and day dreaming I settled on this software stack:

• Kubernetes (obvo)
• Talos Linux driven by Talhelper
• Helm for off the shelf components, driven by Helmfile
• Longhorn for fast replicated storage
• 1Password Operator for secrets management
• Tailscale Operator for private ingress and a subnet router to poke at services and pods directly
• ingress-nginx for internal and external access to services
• MetalLB to give local services a stable virtual IP address
• cert-manager for automatic LetsEncrypt certificates for services
• external-dns to drive DNS updates for services
• Keel.sh for automatic image updates
• Reloader to reload deployments when linked secrets and configs update
• EMQX as the MQTT server for some of my IoT devices (mostly zigbee)
• CloudNative PG for PostgreSQL databases

The next thing to decide was how to divide up the hardware into control plane and worker nodes. Here's what I have so far:

• Three (3) control plane nodes: hypnotoad, crushinator, and lrrr
• Seven (7) local worker nodes: hypnotoad, crushinator, nibbler, shed, three Wyse 3040s hosting Z-Wave sticks
• One (1) cloud worker node

You might notice that several nodes are doing double duty.

Splitting the control plane off to dedicated nodes makes sense when you have a fleet of hundreds of machines in a data center. I don't have that.

A small VM running on Lrrr is the only dedicated control plane node. The only reason for that is because Lrrr also hosts my Unifi and Omada network controllers and I haven't worked up the gumption to move those from Proxmox LXCs to k8s workloads.

Hypnotoad, Crushinator, and Nibbler are general compute. Nibbler has an Nvidia Tesla P4 GPU, which is not particularly impressive but fun to play with. Both Hypnotoad and Nibbler have iGPUs capable of running many simultaneous Jellyfin streams. Crushinator is a VM taking up most of the host which is also serving as a backup NAS for non-media data.

Shed lives in the shed and is connected to a bunch of USB devices, including two SDR radios, a Z-Wave stick, and an RS232-to-USB adapter for the generator.

Morbo is running TrueNAS and has no connection to Kubernetes except that some stuff running in k8s uses NFS shares. It's also the backup target for Longhorn and the script I use to backup Talos' etcd database.

Self-hosting in the Cloud

Talos has a neat feature built in that they call KubeSpan. This is a Wireguard mesh network between all of the nodes in the cluster that uses a hosted discovery service to exchange public keys.

Essentially, you can flip a single option in your Talos configs and have all of your nodes meshed, with a bonus option to send all internal cluster traffic over the Wireguard interface. The discovery service never sees private data, just hashes. It's really cool.

I'm using KubeSpan to put one of my nodes on a VPS to get a public IP without exposing my home ISP connection directly. After initial setup I was able to change the firewall to block all inbound ports other than 80, 443, and the KubeSpan UDP port.

To actually serve public traffic I installed a separate instance of ingress-nginx that only runs on the cloud node. This instance is set up to directly expose the cloud node's public IP which gets picked up by external-dns automatically.

I'm still trying to decide if this single node is enough or if I should get really clever and use a proxy running on Fly to get a public anycast IP.

Ok, but what's running?

Learning how Kubernetes works has been great and this process filled in quite a few gaps in my understanding, but it probably wouldn't have been worth the effort without hosting something useful.

Currently I'm hosting a couple of external production workloads:

• this blog
• VMSave
• a handful of very small websites

I'm also running a bunch of homeprod services:

• Home Assistant
• Whisper and Piper, speech-to-text and text-to-speech tools and components of the Home Assistant voice pipeline
• four (4) instances of Z-Wave JS UI, one per RF "zone" (this house has wacky RF behavior)
• two (2) instances of Zigbee2MQTT, one in each RF zone that has Zigbee devices
• Genmon keeps tabs on our whole home standby generator
• A Minecraft server for me and my kids
• Paperless-ngx stores and indexes important documents
• Ultrafeeder puts the planes flying overhead on a map
• iCloudPD-web for effortless iCloud photo backups
• Jellyfin, an indexer and server for TV shows, movies and music
• Sonarr, Radarr, Prowlarr and SABnzbd form the core of our media acquisition system
• Jellyseerr makes requesting new media easy for the other people who live in the house
• Calibre Web Automated is an amazing tool that serves all of my eBooks to my Kobo eReader
• Ollama and Open Web UI for dinking around with local LLMs
• Homer to keep track of all of the above, set as my browser homepage

Left To Do

There are a few things left on the todo list. Roberto is hooked up to a webcam that watches my 3D printer and I haven't touched it yet because it is connected via Wi-Fi which Talos doesn't support at all.

I also haven't touched the raspberry pi, mostly for the same reason. The pi is serving as a gateway between a Wi-Fi SD card that lives in my CPAP machine and the rest of the network so that I can scrape the data off without having to pull the card or futz with my laptop's Wi-Fi every day.

The Wi-Fi SD card, you see, only exists as an access point. It cannot be put into a mode that connects to another network. The pi has a USB Wi-Fi adapter connected to the card's network and the built-in Wi-Fi connected to the home network with nginx in between serving as a proxy. I don't think this is something that I really want or need to move into k8s.

I want to set up some sort of central authn/authz system for the homeprod services. The current fashion seems to be Pocket ID but I haven't been able to get it working reliably.

I'm thinking about setting up a small ActivityPub server to play around with.

A photo viewer like Immich might be cool to set up.

Overkill?

Of course this is overkill. This could probably all live on a single Wyse 5070 with a couple big harddrives attached.

I think it's been worth it to use Kubernetes in anger. I'm really enjoying the ability to deploy whatever I want to the cluster without having to think about where it runs, where it stores data, etc.

I've also learned a ton and fixed a bunch of preconceived notions and it's already helped increase reliability in a few things that affect household acceptance in big ways.

If you do already have a working style that doesn't involve a web UI and a browser editor you can still use ESPHome, you just have to handle those housekeeping tasks yourself.

Fundamentals

I think it can be helpful to step back and take a look at the fundamentals of a piece of software like ESPHome before diving headlong into the deep pool of non-standard workflows.

At it's base level ESPHome is a microcontroller firmware generator. That is, it reads your YAML config, generates a bunch of C++ files and config files, and then using a compiler and some helper programs it generates a binary program that your microcontroller (usually but not always an ESP32) can run.

ESPHome also has a few very useful helpers. First, it can do seamless over the air (OTA) updates once any ESPHome firmware has been installed on a device.

Second, it has a pretty powerful web-based UI and configuration editor.

Third, ESPHome ships with a "native" binary protocol it can use to talk to Home Assistant (previously) complete with Noise-based symmetric key encryption.

Lastly, it can be used as a Home Assistant addon, which as I said earlier takes care of a few things for you. The OTA update functionality requires a pre-shared key to validate that updates are coming from a known source. The addon takes care of generating that and the Noise secret and sharing these keys with Home Assistant so you don't have to care about them.

ESPHome on a Macbook?!

My ESPHome workflow doesn't involve the web UI or the addon at all. Instead, I install ESPHome on my Macbook with homebrew and manage the OTA and HA secret keys with 1Password and a small helper script. The script and all of my ESPHome configs live in this public GitHub repo.

This is the script as it exists today:

#!/bin/bash
        
        set -x
        set -eo pipefail
        
        trap "rm common/device_base.yaml" EXIT
        
        op inject --in-file common/device_base.yaml.in --out-file common/device_base.yaml
        
        command=$1
        shift
        
        if [ $# -eq 0 ]; then
        configs="*.yaml"
        else
        configs="$@"
        fi
        
        esphome $command $configs
        

All this is really doing is using 1Password's op inject tool to generate a file with my configured secrets, runs esphome, and makes sure to clean up the generated file with that trap line. The top of device_base.yaml.in looks like this:

substitutions:
        wifi_ssid: "op://keen.land secrets/ESPHome Secrets/ESPHOME_WIFI_SSID"
        wifi_password: "op://keen.land secrets/ESPHome Secrets/ESPHOME_WIFI_PASSWORD"
        fallback_ssid_password: "op://keen.land secrets/ESPHome Secrets/ESPHOME_FALLBACK_SSID_PASSWORD"
        home_assistant_encryption_key: "op://keen.land secrets/ESPHome Secrets/ESPHOME_HOME_ASSISTANT_ENCRYPTION_KEY"
        ota_password: "op://keen.land secrets/ESPHome Secrets/ESPHOME_OTA_PASSWORD"
        

All of those are just text entries in the ESPHome Secrets rich text item, but again they can be whatever you want. If you decide to make them password type entries I believe you'd need to add --reveal to the op inject command, but I'm not 100% certain on that.

This differs in kind of a fundamental way from the way the web UI / addon work, in so far as the addon will create and manage unique OTA and HA keys for each device. My setup instead uses two keys shared among all of my devices. I don't see this as a significant risk because I don't use esphome devices in higher security contexts (i.e. my door locks are not running esphome), but your threat model is likely different than mine so you should make your own decisions. Nothing is stopping you from using unique keys for every device with this setup, you just have more secrets to manage in 1Password.

Workflow

My workflow looks like:

$ <edit whatever.yaml>
        $ ./build.sh run whatever.yaml
        # a bunch of compiler output and then logging from the device itself
        $ git add whatever.yaml && git commit -m 'updates' && git push origin main
        

There aren't many hard edges with this setup. You can put whatever you want into common and you can organize your devices however you want.

One exception is that secrets.yaml has some confusing implicit behavior, so I just commit an empty one and use a different file for my secrets.

Updating ESPHome is not something I do on a regular basis, but when I do it's basically just brew upgrade esphome && ./build.sh.

The process to add a new ESPHome device to Home Assistant is also fairly streamlined. All you do is attach the ESP32 device to your computer with USB, create a new yaml file, and run ./build.sh run <your new file>.yaml. ESPHome will pick up that there's a serially attached device without firmware and handle flashing the new firmware to it.

Once ESPHome is running on the new device it should show up in the Home Assistant integrations page and something that can be added. Clicking the accept button will open a config flow where you can paste your Home Assistant key, and then it should work just like any other device.

• Fidelity Brokerage Account ("brokerage") with margin enabled
• Fidelity Roth IRAs
• Fidelity Solo 401K
• Fidelity Multi-Asset Index Fund (FFNOX)
• iShares Core Aggressive Allocation ETF

How Money Flows

• Every investable dollar is in FFNOX across all non-401k tax advantaged accounts
• Every investable dollar in taxable accounts is invested in AOA
• Automatically contribute to my employer's 401K plan every pay period into whatever fund

Why

I don't want anyone to have to think about where to pull money from at any time. I want me or my wife to be able to login to Fidelity and sell enough to cover cash needs with a very small number of clicks.

FFNOX and AOA are funds of funds. FFNOX consists of low cost Fidelity funds and AOA consists of low cost Blackrock iShares ETFs. They both invest in approximately 60% US total stock market, 25% international developed total stock market, and 15% US total bond market. This fits our family's desired asset allocation.

The brokerage account has margin enabled. Margin allows you to borrow up to 50% of the value of your investable assets (everything but cash and CDs) from your broker for any purpose whatsoever. It kicks in while you run out of cash and will automatically pay itself back when you deposit cash in the account.

We have margin turned on so that we don't have to worry about selling investments to raise cash while something awful is happening. We can login to Fidelity and sell some FFNOX when it's convenient rather than having to do it one some kind of schedule.

Contributing to my employer's 401K plan is an automatic tax break and also means I get my employer's matching contribution. I currently (for 2024) have it set up to max out my employer's match.

We don't use robo-investors. FFNOX's total expenses are capped at 0.08%, or $80 per $100,000 annually. Wealthfront charges 0.25% (more than 3x) on top of the fees for the actual ETFs (typically ~0.1%). Robo-investors are never going to offset their fees when compared to FFNOX or similar funds (Target Retirement funds at Vanguard or Freedom Index funds at Fidelity).

My friend Amy Hoy recently tweeted about financial planning and personal finance. This particular tweet stuck out to me:

you'd probably think given my love of money that i'd be all over financial planning, but truly i only enjoy earning it and spending it. saving it feels like… freezing cake. i guess you could do it, but why, you have cake

— Amy Hoy ✨ (@amyhoy) February 19, 2019

In early 2018 I radically simplified my family's personal financial system and made everything as automatic as possible. Amy and Joel Hooks asked me to write up how it works, so this is the start of a series of short posts about how and why I set everything up.

A Little Bit of Why

I prefer to think of myself as a realist. Due to my health history my wife is likely going to be around longer than me. Her family has some very long lived female members as well. Her grandmother is 103 and her great aunt just passed in 2018 at the ripe old age of 95.

I want to make our finances as simple as possible so she doesn't have to worry about them when the inevitable happens.

In 2016 my wife and I welcomed our first child into the world and in late 2018 we welcomed our second. They are two more very important reasons why I want things to be simple. If something happens to both my wife and me, I want our intentions with regards to our finances as plain as possible.

This system got a trial run in late 2018. My wife was admitted to the hospital at 29 weeks pregnant for preeclampsia, a very dangerous condition that needs close monitoring. My daugther was born at 34 weeks and spent the next five weeks in NICU.

I didn't have to touch this system at all. Not once. I logged in a handful of times to check up on it, but everything just hummed along.

• One Fidelity Brokerage Account ("brokerage")
• One Fidelity Cash Management Account ("CMA")
• One Fidelity 2% Cash Back Credit Card
• One Chase Freedom Unlimited Credit Card
• One Amazon Prime Visa

How Money Flows

• Payroll direct deposited into brokerage
• All non-mortgage recurring expenses are paid with the Fidelity card
• All Amazon expenses are paid with the Amazon card
• All other expenses are paid with the Chase card
• Bills autopay from brokerage (credit cards, insurance, billpay to household vendors)
• Debit cards and live paper checks written against CMA

Why

We use a brokerage account because it lets us keep cash and investments in the same account. All of our cash, including working capital and reserves, sits in the brokerage's core position. Our core position is FZFXX, a Federal money market fund that pays ~5% interest as of July 2024.

We use the Fidelity credit card because it pays 2% cash back when it's set up to deposit rewards into a Fidelity account. It is also one of the only cards I've seen that can be set to automatically cash out deposits. Ours is set to deposit into the brokerage account. We only use this card for recurring expenses. Non-recurring expenses go on the Chase card because in our experience Chase's fraud detection and customer support is considerably better than Fidelity/Elan.

We use the Amazon Prime credit card for all Amazon expenses. This pays 5% back on Amazon purchases which makes it worth it for us. Your milage may vary.

We have the CMA so we don't expose the brokerage account number every time we write a paper check. This is probably overly paranoid and is the only significant complication in the entire system.

The CMA has "self-funded overdraft protection" turned on which automatically transfers from the brokerage account into the CMA to fund checks and debit card transactions.

The brokerage core position is not FDIC insured. I don't care about FDIC insurance. FZFXX is composed of ultra short term US Treasury bills. If Treasuries are suddenly not liquid enough to withdraw our money our society has much bigger problems.

The CMA's core position is FDIC insured, and the CMA is almost a full brokerage account, but we don't use it as the centerpiece account for two reasons. First, the CMA core position pays shit for interest. Second, the CMA cannot have margin turned on. We'll talk about why that's important in the next post.

After using it for a season I discovered that the reason why this controller is so cheap is because it uses an ESP8266, which is fine, but it doesn't play well with my setup. For example, if I enable the Home Assistant integration the controller falls over after a few hours. It also reboots for unknowable reasons sometimes and I would come home to find the lights in their default orange color.

I probably could have fixed this with a more powerful controller. I even bought a neat ESP32 controller with a built-in wide range voltage regulator but never got around to setting it up.

Loosely, what I want is:

1. Control the lights without random reboots
2. Easy Home Assistant integration
3. Easy customization
4. No Wi-Fi
5. Use hardware that I already have
6. Tailscale, ideally
7. Learn some stuff

I could have gotten the first two using a more powerful ESP32 module. Third could be done with ESPHome. Four and five are contradictory while staying within the constraint of an ESP32-based system.

Also last year, I built a little box that controls the power for my 3D printer with a Raspberry Pi Pico and a second Klipper instance (previously) so naturally I tried to get Klipper to fit in this non-3d-printer shaped hole. I tried so hard.

On the surface, Klipper appears to do everything that I want (control addressable LEDs, kind of customizable) but it makes no compromises in wanting to be a 3D printer controller. Most of the firmware is dedicated to running a motion controller, there's a lot of emphasis on scheduling things to happen in the near future, and there's a global printer object. Importantly for my purposes, there's no built-in way to set up a digital input without declaring it a button.

It's fine. Klipper is fine. It's just not built to be a generic IO platform.

So, what's a reasonable rational person to do?

Write an ESPHome Protocol Server

Of course.

There are essentially three ways to get arbitrary devices and entities to show up automatically in Home Assistant.

First, one can write a Home Assistant integration. This is fine and good but it doesn't work for me because my devices are far away from the VM that Home Assistant runs in.

Second, there's MQTT autodiscovery. I know this works because it's how my Zigbee devices integrate with HA, but I just could not make any of the existing generic autodiscovery libraries work consistently. Usually I would end up with a bunch of duplicate MQTT devices and then HA would get confused.

Third, there's ESPHome. ESPHome is a firmware for ESP modules (think: small devices with wifi like plugs, air quality monitors, etc). ESPHome belongs to the Open Home Foundation, same as Home Assistant, so it has commercial support and a first class HA integration. I already have a bunch of ESPHome devices running in my house, so it seems like a pretty natural fit.

The normal and ordinary way of using ESPHome is to write some YAML config that ESPHome compiles into a firmware for your device, then you flash the device and HA sets itself up to interact with the entities you described in YAML. What I want to do is just that last bit, the part where I can tell HA what entities I have and it sets up UI for me.

HA talks to ESPHome over what they call their "native API". The native API is a TCP-based streaming protocol where the ESPHome device is the server and Home Assistant is the client. They exchange protocol buffer encoded messages over either plain TCP or with a Noise-based encryption scheme.

Over the last week or so I built a Python implementation of that protocol named aioesphomeserver, bootstrapping off of the official aioesphomeapi client library that HA uses.

A Minimal Example

Here's a very simple example of what aioesphomeserver looks like:

import asyncio
        
        from aioesphomeserver import (
        Device,
        SwitchEntity,
        BinarySensorEntity,
        EntityListener,
        )
        
        class SwitchListener(EntityListener):
        async def handle(self, key, message):
        sensor = self.device.get_entity("test_binary_sensor")
        if sensor != None:
        await sensor.set_state(message.state)
        
        device = Device(
        name = "Test Device",
        mac_address = "AC:BC:32:89:0E:C9",
        )
        
        device.add_entity(
        BinarySensorEntity(
        name = "Test Binary Sensor",
        )
        )
        
        device.add_entity(
        SwitchEntity(
        name = "Test Switch",
        )
        )
        
        device.add_entity(
        SwitchListener(
        name="_listener",
        entity_id="test_switch"
        )
        )
        
        asyncio.run(device.run())
        

From the top, we import a bunch of stuff and then create a class that listens for messages from the device (the handle method). Then, we set up a device with a name and a fake MAC address. Device can generate a random one for you but it doesn't persist, so if you want this device to stick around in HA you should declare a static MAC.

We then add some entities to it: a binary sensor, a switch, and an instance of our switch listener configured for Test Switch.

Finally, we start the asyncio event loop.

With just that, you get the ESPHome web UI:

Adding the device to Home Assistant you'll see this:

AIO ESPHome Server Architecture

I tried to follow the spirit of ESPHome's architecture when writing the server.

The Device is a central registrar for Entitys and serves as a message hub. The native API server and web server are entities that plug into the message bus, as are things like SwitchEntity and BinarySensorEntity. Everything is async using Python's asyncio.

Any entity with a run method will automatically be scheduled as a task at startup.

A Production Example

The development case for this library has been driving the addressable LEDs on my house. I found a project named u2if that turns a Raspberry Pi Pico into a USB peripheral that provides a bunch of fun stuff: GPIO, I2C, SPI, PWM, ADC as well as an addressable LED driver for WS2812-compatible lights. The fun wrinkle of the light driver is that it offloads the bitstream generation to the Pico's PIO coprocessors.

I forked u2if and added a few things:

• RGBW support, which was already in the codebase but not available
• Support for the Pico clone boards I have (SparkFun Pro Micro RP2040)
• A set of effects for Neopixel along with a console-mode simulator to use while developing
• A Docker image that bundles the firmware and the Python library

This deployment consists of:

• A Dell Wyse 3040 thin client running Alpine Linux that already handles Z-Wave for the garage
• SparkFun Pro Micro RP2040 running the u2if firmware connected over USB
• Two channels of RS485 transceivers so I can get the very fast, very unforgiving light control signals 40 feet from where the 3040 is mounted to the wall to where the light power injector lives.

Here is the full script that I'm using to drive the addressable lights on my house:

import asyncio
        
        from machine import WS2812B
        from neopixel.effects import StaticEffect, BlendEffect, TwinkleEffect
        
        from aioesphomeserver import (
        Device,
        LightEntity,
        LightStateResponse,
        EntityListener,
        )
        
        from aioesphomeapi import LightColorCapability
        
        class LightStrip(EntityListener):
        def __init__(self, *args, strings=[], effects={}, **kwargs):
        super().__init__(*args, **kwargs)
        self.strings = strings
        self.num_pixels = sum([s[1] for s in strings])
        self.effects = effects
        
        self.current_effect_name = None
        self.current_effect = StaticEffect(count=self.num_pixels)
        self.white_brightness = 0.0
        self.color_brightness = 0.0
        
        async def handle(self, key, message):
        if type(message) != LightStateResponse:
        return
        
        await self.device.log(1, f"message.effect: '{message.effect}'")
        
        if message.effect != "" and message.effect != self.current_effect_name:
        if message.effect in self.effects:
        self.current_effect_name = message.effect
        self.current_effect = self.effects[message.effect](self.num_pixels, message)
        self.current_effect.next_frame()
        
        if self.current_effect:
        self.current_effect.update(message)
        
        self.color_brightness = message.color_brightness
        self.white_brightness = message.brightness
        
        if message.state == False:
        self.color_brightness = 0.0
        self.white_brightness = 0.0
        
        def render(self):
        pixels = []
        
        for i in range(self.num_pixels):
        color = self.current_effect.pixels[i]
        
        pixel = [
        int(color[0] * 255.0 * self.color_brightness),
        int(color[1] * 255.0 * self.color_brightness),
        int(color[2] * 255.0 * self.color_brightness),
        int(color[3] * 255.0 * self.white_brightness),
        ]
        
        pixels.append(pixel)
        
        # partition strings
        # write to each string
        cur = 0
        for string, length in self.strings:
        last = cur + length - 1
        string.write(pixels[cur:last])
        cur = last + 1
        
        async def run(self):
        while True:
        self.current_effect.next_frame()
        self.render()
        await asyncio.sleep(1/24.0)
        
        
        device = Device(
        name = "Garage Stuff",
        mac_address = "7E:85:BA:7E:38:07",
        model = "Garage Stuff"
        )
        
        device.add_entity(LightEntity(
        name="Front Lights",
        color_modes=[LightColorCapability.ON_OFF | LightColorCapability.BRIGHTNESS | LightColorCapability.RGB | LightColorCapability.WHITE],
        effects=["Static", "Twinkle"],
        ))
        
        def make_twinkle_effect(count, state):
        return BlendEffect(
        TwinkleEffect(count=count, include_white_channel=True),
        StaticEffect(count=count, color=[state.red, state.green, state.blue, state.white], include_white_channel=True),
        mode='lighten',
        include_white_channel=True,
        )
        
        device.add_entity(LightStrip(
        name = "_front_lights_strip",
        entity_id = "front_lights",
        strings = [(WS2812B(23, rgbw = True, color_order="GRBW"), 20)],
        effects={
        "Static": lambda count, state: StaticEffect(count=count, color=[state.red, state.green, state.blue, state.white], include_white_channel=True),
        "Twinkle": make_twinkle_effect,
        },
        ))
        
        asyncio.run(device.run())
        

The structure is basically the same as the minimal example. We import some stuff, we set up an EntityListener class, and then we set up a Device with a LightEntity and an instance of the listener .

In this case, the listener listens for state responses from a Light entity and renders pixels according to a set of effects. It also has a run method that renders the current effect out every 1/24th of a second.

Should you use this?

I don't know!

If your constraints match mine, maybe it'd be helpful. If you want to expose a thing to Home Assistant and would rather have it show up as an ESPHome device rather than, say, writing your own HA integration or messing with MQTT or writing RESTful API handlers, this would probably be useful.

That said, I think if your use case fits within ESPHome proper you should use that. ESPHome has built in drivers for so many things and is going to be better supported (i.e. people are paid to work on it).

Pretty neat though, eh?

Until this past Monday, when I installed some grass seed and couldn't promise myself that I'd water it twice a day every day for the next month as one is supposed to do.

A normal person would just buy a mechanical or stand-alone timer device and be done with it. Not me. No way.

My first thought (a completely normal one, mind you) was to grab an esp32 and a relay board out of my parts bin and liberally apply ESPHome to the problem. This didn't happen because I recently cleaned my office and put those parts in a bin that I can't quite find right now.

cough. Anyway.

My fallback plan was to grab a Z-Wave relay that was more readily at hand and just ("just") use Home Assistant as the timer.

Parts List

• Zooz Z-Wave ZEN17 Universal Relay
• Orbit 3/4" inline sprinker valve
• 3/4" NPT to garden hose thread adapters
• Generic 24VAC transformer
• 5 position terminal block
• 18AWG stranded two conductor silicone jacketed wire
• some random crimp on butt connectors I had laying around
• A sprinkler

In-ground sprinkler valves in the US pretty universally use 24V AC power. This is a very common signaling voltage for HVAC and other controls in residental settings so it makes sense. For example, non-smart doorbells are also generally 24-ish VAC.

The neatest thing about the Zooz ZEN17 (and it's friend the ZEN16) is that it can be powered by USB-C or 12-24 volt DC or AC. That means I can get away with just one power supply for the whole contraption without having to get any kind of fancy voltage regulator.

Sprinker valves come in a variety of sizes, none of which are directly compatible with garden hoses. In the US and Canada residental garden hoses have a universal connector size and thread: 3/4" NH (national hose) (or GHT (garden hose thread)), otherwise known as ANSI B1.20.7. So, I needed some thread adapters. Your local hardware store or supermegaglobal online bazaar named after a river in Brazil will provide you with many options.

Assembly

I didn't take any good pictures of the prototyping and assembly process. Hopefully these will be sufficient.

As you can see, the power supply is connected directly to the relay (via a small terminal block affixed to the relay with a command strip) and the red and black wires lead to the valve. One of the valve wires is connected directly to the power supply and the other is connected to the NO (normal open) terminal on one of the relays in the ZEN17. A short jumper wire is connected from the other leg of the power supply to the C (common) terminal on the same relay.

Here's a shot of the valve itself. You can see the green valve body, the black solenoid that activates the valve, and the brass thread adapters. You can also see the craptastic butt splice things I used, which will be replaced with proper waterproof junctions as soon as they arrive.

The grey and orange piece on the right is a quick release adapter. I have quick releases on the spigot and all of the hoses, which lets me easily install and remove the valve as needed.

Home Assistant

A generic relay isn't much good without something to control it, so let's turn to Home Assistant.

The central component of the sprinkler controller is a 5 minute timer helper which ties with two automations:

1. Sprinkler Start triggers at 8:30am and 5:30pm and starts the timer, but only if the forecast precipitation probability from OpenWeatherMap is less than 50%.

2. Sprinkler Relay triggers whenever the timer changes state.

Sprinkler Relay uses a Choice action with two branches:

• Timer active and relay off -> turn the relay on
• Timer idle and relay on -> turn the relay off

I also have a Time Pattern trigger on Sprinkler Relay that runs every minute. This acts as sort of a failsafe, so that if the timer ends during a Home Assistant restart, the sprinkler will shut off with at most an extra minute of runtime.

Here's Sprinkler Start in YAML:

alias: Sprinkler Start
        description: ""
        trigger:
        - platform: time
        at: "08:30:00"
        - platform: time
        at: "17:30:00"
        condition:
        - condition: numeric_state
        entity_id: sensor.openweathermap_forecast_precipitation_probability
        below: 50
        action:
        - service: timer.start
        metadata: {}
        data: {}
        target:
        entity_id: timer.sprinkler_runtime
        - service: notify.all_phones
        metadata: {}
        data:
        message: Sprinkler has started
        title: Sprinkler
        mode: single
        

and here's Sprinkler Relay:

alias: Sprinkler Relay
        description: ""
        trigger:
        - platform: state
        entity_id:
        - timer.sprinkler_runtime
        - platform: time_pattern
        minutes: "*"
        condition: []
        action:
        - choose:
        - conditions:
        - condition: state
        entity_id: timer.sprinkler_runtime
        state: idle
        - condition: state
        entity_id: switch.universal_relay_1
        state: "on"
        sequence:
        - service: switch.turn_off
        metadata: {}
        data: {}
        target:
        entity_id: switch.universal_relay_1
        - conditions:
        - condition: state
        entity_id: timer.sprinkler_runtime
        state: active
        - condition: state
        entity_id: switch.universal_relay_1
        state: "off"
        sequence:
        - service: switch.turn_on
        metadata: {}
        data: {}
        target:
        entity_id: switch.universal_relay_1
        mode: single
        

So, what do I have after all of that? I have a sprinkler valve that could be automated with a wrist watch but is actually automated by a Rube Goldberg-esque stack of technology, including radio protocols, ethernet, virtual machines, Python, Javascript, and Tailscale (because of course).

Simple!

Home Assistant doesn't have a direct way to read JSON data from a file into a sensor. There's the File platform which has a promising name but is actually a trap. File is meant for use cases where something writes to, say, a CSV file continuously and you just want to read the most recent line. It specifically does not read the whole file.

After a lot of searching I came across the Command Line platform. The integration does a number of things, but for our purposes it lets you periodically run a command within the context of the Home Assistant container and bring the output back into Home Assistant as a sensor.

Let's say you have a JSON file named rate.json in your Home Assistant configuration directory:

{
        "name": "Base Rate",
        "rate": 0.15
        }
        

You can bring that into a sensor with the following snippet in your configuration.yaml file:

command_line:
        - sensor:
        name: "Electrity Rate"
        command: 'cat rate.json',
        value_template: "{{ value_json['rate'] }}"
        unit_of_measurement: "USD/kWh",
        json_attributes:
        - name
        - rate
        

This config does a couple things. The command key specifies what command HA should run, in this case cat to read the file to stdout. value_template extracts the rate key from the file into the sensor's value. The json_attributes list pulls the list of keys into attributes in the sensor, which you can later access from a template using state_attr(). I have also specfied unit_of_measurement here just because the Energy reporting system needs that if you want to use this as an input.

So, the above is great if you have one static set of attributes to bring in, but sensor values can be at most 255 characters. What if you have a bigger file that you need to pull just a little data out of?

Let's say we have this slightly bigger file rates.json:

[
        {
        "id": "d1-11_summer_on_peak",
        "name": "Summer Peak",
        "months": [6, 7, 8, 9],
        "days": [1, 2, 3, 4, 5],
        "hours": [15, 16, 17, 18],
        "rate": 0.23525,
        "peak": true
        },
        {
        "id": "d1-11_summer_off_peak",
        "name": "Summer Off-Peak",
        "months": [6, 7, 8, 9],
        "days": [0, 1, 2, 3, 4, 5, 6],
        "hours": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "rate": 0.17859,
        "peak": false
        },
        {
        "id": "d1-11_winter_on_peak",
        "name": "Winter Peak",
        "months": [1, 2, 3, 4, 5, 10, 11, 12],
        "days": [1, 2, 3, 4, 5],
        "hours": [15, 16, 17, 18],
        "rate": 0.17879,
        "peak": true
        },
        {
        "id": "d1-11_winter_off_peak",
        "name": "Winter Off-Peak",
        "months": [1, 2, 3, 4, 5, 10, 11, 12],
        "days": [0, 1, 2, 3, 4, 5, 6],
        "hours": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "rate": 0.1658,
        "peak": false
        }
        ]
        

This file describes the rate structure that's in effect at my house, DTE rate D1.11. Each entry in the array is a rate, and the first matching rate based on month, weekday, and hour is the effective rate.

Ordinarily one might reach for a HACS integration or something, but the Home Assistant container has another trick up it's sleeve: it bundles jq.

jq is a tool for querying and manipulating JSON streams. This essay isn't meant to be a jq tutorial so we're not going to go in depth into what this query in rate_filter.jq does, but in broad strokes it picks the first matching rate from the input file and extracts just the name, rate, and peak keys.

map(select(
        (.months[] | contains($ARGS.positional[0] | tonumber))
        and (.days[] | contains($ARGS.positional[1] | tonumber))
        and (.hours[] | contains($ARGS.positional[2] | tonumber))
        ))[0] | {name, rate, peak}
        

Here's a modified command line sensor that runs jq appropriately:

command_line:
        - sensor:
        name: "DTE Rate"
        command: 'jq -f rate_filter.jq rates.json --args {{ now().month }} {{ now().isoweekday() % 7 }} {{ now().hour }}'
        value_template: "{{ value_json['rate'] }}"
        unit_of_measurement: USD/kWh
        json_attributes:
        - name
        - rate
        - peak
        

The relevant change here is to the command key, which now invokes jq with the -f argument to pass the filter as a file rather than trying to quote everything properly within HA, then passes the actual rates.json file, then treats the rest of the arguments as positional args. These are accessed within rate_filter.jq as $ARGS.positional[0] etc.

With this set up I can access the current electric rate within my Home Assistant in a way that is compatible with the Energy dashboard, which is completely local, and which should be easy to maintain in the future.

The above isn't at all specific to electric rates, by the way. This technique should work for any data that you need in HA but is more complicated than a plain input can work with.