Homelab · Ethan Justice

The map

Architecture

This is what runs on it: 7 guests (3 VMs, 4 LXC containers) carrying 31 Docker containers. Atlas, my private voice for alerts, and Jarvis, a public bot, live here, one trusted and one deliberately not, and so does my Mealie fork. One GPU does four jobs. The security controls in front of all of it have their own page.

In one line: everything reaches the internet through one hardened front door, and the apps that must stay private are blocked from every other way out. Run the scenarios below to watch it, or click any box for what it runs and why.

homelab · traffic map trace it

Pick a scenario and watch the packet move.

Pull the plug, then run "Locked to the VPN" again.

Click a box, or focus it and press Enter.

Select a component, or trace a path, to see how egress is controlled.

Internet

Two egress paths reach it, and they are not interchangeable. Ordinary services leave through the LAN gateway. Isolated workloads can only reach it through the VPN provider, because the firewall inside their namespace drops everything else.

VPN provider endpoint

The single internet egress for the isolated namespace. Anything else headed for the internet is dropped before it leaves the namespace, so this is not the preferred path out. It is the only one allowed.

LAN gateway

The normal route to the internet for ordinary services. The isolated workloads cannot use it to get out: their namespace's firewall drops by default and accepts only the tunnel, the VPN handshake, and the local subnet the media stack talks to.

Hypervisor · Proxmox VE

One host, carved into VMs and LXC containers. Every guest carries its own default-deny firewall rather than relying on the cluster firewall, so the host is not the only thing between a service and the network, and guests stay portable.

Container host (LXC)

Runs most of the containerized apps. The ones that must only ever use the VPN share one network namespace with the VPN container instead of having a network of their own.

VPN tunnel

Everything the isolated workloads send to the internet leaves through here. If it drops, they do not fall back to the home connection, they lose the internet until it comes back, which is the correct failure mode. I killed it six times to check: 45 probes during the outages, none answered, and it recovered on its own in 23 to 61 seconds.

Isolated workloads

Containers that need internet access but must never use the home connection. They share the VPN container's network namespace, where the firewall's default is drop. A leak would need a rule that is not there, and I check that by killing the tunnel, not by reading the config.

Proxy VM · Caddy

All TLS terminates here and every service sits behind it. Nothing is exposed directly, so there is one front door to harden instead of a dozen.

Requests LXC

The user-facing intake, kept separate from the workloads that act on what it receives. A blast-radius decision: the part strangers touch holds the API keys it needs to pass a request on, but has no media library and none of the apps that act on requests.

Media VM · GPU passthrough

The RTX 3070 is passed through to this VM alone and does four jobs: NVENC transcoding for playback, speech for the voice stack, a local language model for translation, and image upscaling. Transcoding wins: the upscaler pauses whenever a stream holds an NVENC session, and waits for the card to cool from 75 °C to 65 °C before it continues.

Playback · NVENC

Hardware transcoding on the 3070. This is the job the GPU was bought for. Everything else on the card is batch work that gets out of its way.

AI jobs · speech, translation, upscaling

Whisper for speech-to-text and Kokoro for the voice, a local model for translation, and an upscaler. All of it runs on the same card, with no cloud API in the loop. Measuring the speech pipeline is what showed that text-to-speech, the stage that sounds slow, is the fastest part of a response.

Atlas · the private voice

Home Assistant runs the house, in its own container. Atlas is the voice on top: a watcher on the hypervisor reads health, security and self-heal state every 60 seconds, and Home Assistant decides what gets said and where, on my PC when I'm at it, otherwise my phone. Only I hear it, which is why it is allowed to know all of this.

Jarvis · the public bot

The Discord bot my friends talk to. It runs on the container host with 18 tools (music, polls, timers, web search, media requests), and none of them touch the house: there is no Home Assistant credential in its environment and no Home Assistant code in its image.

Mealie · my fork

A fork of the recipe manager that I build and deploy here, with an acceptance check on every change. Running what I build is the point: every regression I ship lands on the machine I use at dinner.

Dashboard LXC

Six pages that lead with a one-line verdict: health, security, media, books, a monthly tool radar, news. Acknowledgements happen on the page with an undo, and actions are signed links. My phone reads a summary of it over a private tunnel.

Monitoring · Netdata, watchdog, self-heal

Netdata keeps per-second metrics. On top of it, a watchdog checks that 168 jobs and feeds are still updating, and 72 health probes repair what they safely can. Both are explained below. An AI assistant can read all of it through read-only tokens; any change it proposes waits for my go.

ZFS storage · three pools

Guest disks live on a two-disk mirror, the media library on a second mirror. Nightly backups go to a separate disk and a weekly copy goes to the media mirror, so one dead disk never takes a guest and its only backup together.

Watchdog

Nothing gets registered by hand

In one line: if anything stops updating, I hear about it before I would ever notice it myself. Monitoring usually answers "is it up?". The failure I care about more is quieter: a scan that stopped running, a backup that stopped landing, a dashboard that shows last Tuesday as if it were today. Every 10 minutes a watchdog finds every scheduled job on all eight hosts and every data feed the dashboard reads, by itself, and checks each one is still updating. Right now that is 168 checks, 168 of them fresh.

The limit comes from the schedule. A timer or cron job goes stale at twice its interval plus a margin, so a job that runs every 6 hours is late after 12 h 10 min, not at a number I picked. Feeds with a known producer get a limit that matches it.
New jobs are covered the moment they exist. A data feed the watchdog cannot match to a job is itself flagged: "no refresher registered". That is how it finds gaps instead of waiting for me to remember them.
It repairs only what is safe to repeat. A stale feed with an idempotent generator is re-run once, at most every 6 hours, before anyone is alerted. Everything else goes straight to an alert. It never restarts or rebuilds anything.

watchdog · sample run fast-forward it

clock +0 h 00 min

Sample jobs with the watchdog's real limit rules. Names are generic; the rules are the ones in production.
Job or feed	Runs	Stale after	State
Dashboard health feed	every 10 min	30 min	fresh
Self-heal status	every 10 min	25 min	fresh
Container vulnerability scan	daily	26 h	fresh
Hypervisor config backup	daily	26 h 30 min	fresh
Backup integrity check	weekly	8 days	fresh

Recovery

Backups, timed

In one line: every guest is backed up twice, to different disks, and restores are drilled on a stopwatch. A backup that has never been restored is a hope.

Backup layers, read from the live job configuration.
What	When	Where	How
Every running guest	Daily 04:00	Separate backup disk	Snapshot mode, zstd
Every running guest, second copy	Sunday 05:30	Media mirror (different disks)	Snapshot mode, zstd
Hypervisor config and scripts	Daily 03:17	Backup disk	7 rolling + monthly
Archive integrity check	Sunday 08:41	Both targets	Newest archive per guest, alert only on problems
Backup age	Every 10 min	Watchdog	Alert if any guest's newest backup is over 8 days old

restore drill · 2026-10-22

SIEM VM: 39 s

Restored with its network unplugged; every SIEM component came up.

Timed from the hypervisor task log 19:20:46 to 19:21:25.

restore drill · 2026-10-30

Request-intake container: 1 min 35 s

Database integrity ok; users, requests and media rows identical to live (7 / 226 / 611).

Timed from the hypervisor task log 14:15:15 to 14:16:50.

Both drills restore into a throwaway copy with no network, so a restored service can never answer for the live one. Next drill: boot the restored applications on an isolated network, not only check their data.

Postmortems

Three things that broke, and what changed

Each looked fine until it didn't. For each: what was wrong, how I found it, the fix, and the proof the fix holds.

memory · hypervisor · 10-26 to 10-29

The swap that kept coming back

The usual fixes treated the symptom. Per-container counters found the one app causing 86 % of it.

Wrong: Host swap hit 98.5 % with 16.7 GB of RAM free. Three textbook fixes (a smaller ZFS cache, more RAM for the container, swappiness 1) each held for about two hours.
Found: Per-cgroup counters instead of host totals: one container caused 86 % of all swap-outs. Inside it, one file-heavy app read its data through memory mapping, which parked 7.24 GiB of file pages on that container's memory bill, so the kernel pushed every other service out to swap.
Fix: Plain file I/O for that app, kernel watermark boosting off, a swap cap on the container, and a logger that records swap per guest every 10 minutes.
Proof: That app dropped from 7.3 GiB to 63 MB the moment the setting changed, and swap stayed flat overnight. The logger has since pointed at a second, smaller source, the SIEM VM, which is the next fix.

self-heal · 10-22

Self-heal rebooted the wrong VM

Automation that guesses is dangerous. Now it checks who really owns an address before it acts.

Wrong: A TV on the network grabbed the media VM's address. The probe saw the media server stop answering and rebooted the proxy VM, which was fine. During a planned reboot, the same script "repaired" 20 services that were only shutting down.
Found: The outage itself, then the repair log line by line.
Fix: Version 2 sends raw ARP requests to see who answers for each static address and raises an IP-conflict alert instead of acting. It reboots the proxy only when SSH to it fails, does nothing until the host reports itself up and has been up 10 minutes, and makes at most 3 repairs in 6 hours.
Proof: A container stopped on purpose on 10-30 was repaired in about 1 min 45 s, and Atlas announced it. 72 probes run every 10 minutes, one pass in 27 s.

disk-pressure cleanup · 10-30

A delete button on autopilot

Caught a cleanup tool a month before it would have started deleting the wrong things.

Wrong: The tool that frees space by deleting media was in automatic mode, blind to watch history since a media-server upgrade, and about a month from its first real run. Over 700 GB of library files were hardlinks to older copies, so a delete would free nothing and the next pass would delete more.
Found: An audit of its source, database and settings, before it had deleted anything.
Fix: Approval mode, so nothing goes without a click. A small internal shim that translates its auth header for the new media server. 2.1 TB of orphaned files moved aside with a restore manifest. An upstream bug report for a release that broke its own login page.
Proof: The delete path, proven end to end on a hardlinked test title: approval, 30-second queue, removal, import exclusion, audit entry. The media-server integration recovered after 9,266 failed calls.

Decisions

Design decisions

Each of these was a choice with a reason, not a default I inherited.

Guests on a mirror, backups on other disks. One dead disk can never take a guest and its only backup together.
Restores are drills, with a stopwatch. 39 s for the SIEM VM, 1 min 35 s for the request database.
Default-deny nftables inside each guest instead of the cluster firewall. The guests stay portable, and the host is not the only layer.
Egress isolation that fails closed, and is tested. The workloads that must not leak share the VPN container's namespace, whose firewall drops anything not headed into the tunnel. Six tunnel kills, 45 probes while it was down, zero answers.
TLS terminates at one hardened proxy. Services are never exposed directly. What guards it.
Automation repairs only what is safe to repeat. Self-heal and the watchdog alert on everything else, and anything destructive waits for a human.
Updates roll themselves back. Weekly image updates pass a health gate or return to the previous image; I proved that path by forcing a failed gate on a real service before trusting it.