Every control, attacked

In one line: four layers stand between the internet and any app, and each was tested from outside.

Everything public sits behind one reverse proxy VM. A request has to clear four layers before it reaches an app: a geo filter in nftables covering TCP 443, UDP 443 (HTTP/3) and TCP 80; a CrowdSec ban list enforced in both the firewall and the proxy; a WAF running CrowdSec virtual patches and the OWASP Core Rule Set in blocking mode; and TLS termination with credentials redacted before anything is written to a log. Since 10-31 the proxy also adds baseline security headers to every public app that sent none of its own (nosniff, referrer and permissions policies, frame protection) and strips the server banners that named the software behind it.

I sent ten requests through the edge's WAF engine, using the same API the proxy calls, six times each. Every attack payload was blocked with a 403 before it could reach an app, with the same result in all six runs. The table has every request, verdict, score and rule. Watch one go through the layers on the home page.

The one request that is allowed asks for a WordPress login page. By itself that is not an attack payload; scanning for it is, and CrowdSec's wordpress-scan scenario catches that pattern from the proxy logs. Four blocked requests from one source earn a 24-hour ban, and the firewall starts dropping that source 7.0 to 10.1 seconds after the decision (three measured runs).

How the WAF got to blocking: the Core Rule Set ran in detection-only mode first, scoring real traffic without blocking it. It flagged none of about 5,300 real requests, so on 10-30 it moved to blocking. Detect, tune, then enforce is the order I would want at work too.

The tuning was not finished. Three days after the switch, the media app on my phone crashed every time I changed video quality. Its crash log showed an unhandled 403: the app stops the old stream with an HTTP DELETE, and the Core Rule Set rejects DELETE, PUT and PATCH by default. The same rule was blocking recipe edits and request changes from outside the house. My 48-hour false-positive check had only counted blocks in the logs, and nobody had edited or deleted anything from outside in that window, so it found nothing. The fix exempts that one method rule for those three methods; every injection rule still inspects them, which the re-test confirmed (an SQL injection sent as a DELETE is still blocked). The lesson: a false-positive check has to send each app's write calls itself, not wait for real traffic to find them.

Each of these looked fine in the configuration and broke the first time I attacked it from the outside. All four are fixed and re-tested. For each: what was wrong, how it showed up, the fix, and the re-test.

In one line: if something gets past the edge, every host still denies by default, is hardened, and reports to a SIEM.

Every guest runs its own default-deny nftables table, loaded by its own service, so the hypervisor is not the only thing between a service and the network. Admin ports are reachable from three hosts. It ran in audit mode for 46 hours before enforcing, and the enforcing rollout fired zero auto-reverts.

All eight hosts are hardened against the CIS benchmarks in two tiers: SSH key-only for root, account lockout, audit rules, password length over composition rules (NIST SP 800-63B). Wazuh's configuration assessment scored every host 40 to 43 before and 51 to 55 after.

Wazuh collects from all eight hosts. On 10-31 I planted a file in the systemd unit directory of every one of them; all eight raised an alert within 2 seconds. On the hypervisor I planted more on 10-30: touching sudoers raised a level 10 alert, running a binary from /tmp a level 12, and an EICAR test file a level 12 VirusTotal hit (66 of 68 engines). The tuning goes the other way too: a monitoring agent's daily updater runs a script from /tmp, so one rule matches only that exact command shape. The same shape with any other script name still fires level 12.

Trivy scans every container image daily. Most critical CVEs in public images need the upstream project to rebuild, so a weekly reminder became noise. The alert is now keyed to the exact image and CVE pairs: fewer CVEs stay quiet, a new one alerts at once. Updates and automatic rollbacks are on the homelab page.

I scanned every commit of my four public repositories with gitleaks 8.28.0: 9,194 commits, 245 of them mine. Findings in my commits: 0. The other 24 findings are test fixtures and saved recipe pages in the forked projects' own history. This site has its own gate: a build fails if a private word or an inline script reaches the output.

In one line: every control above reports to one place, and only what needs me reaches me.

A read-only collector pulls from CrowdSec, the WAF, Wazuh and the image scanner every five minutes. What it finds is shown in levels, the way plant control rooms do it (ISA-101): a phone widget that answers "is anything wrong", a dashboard with one-line verdicts and a short "needs attention" list, the SIEM's own dashboard for level 10 and up, then the raw events. Each item carries a signed link to acknowledge it, undo that, or ban the source. An acknowledged item still counts in the totals; it just stops turning the tiles red. Test events get acknowledged only after the audit trail proves where they came from.

That routing came from getting it wrong. On 10-27 at 00:51 the assistant woke me through Do Not Disturb to report a single probe hit. Over the next three days: low-hit probes became visual only, speech got a quiet window overnight unless the alert is loud, it speaks where I am (a presence heartbeat from the PC decides desk or phone), and the phone checks Do Not Disturb live instead of trusting a sensor that could be minutes stale.

The edge also gets tested from outside every day, not only when I think of it. A scheduled job asks Globalping, a public network of probes, to load the site from probes inside the allowed region (the control) and from probes outside it. Outside probes must time out at the firewall; inside probes must get an answer; any outside answer is spoken as a failure. First run, 11-02: 3 of 3 inside probes answered, 8 of 8 outside probes in eight countries timed out, on HTTPS and on port 80. The probe network cannot test HTTP/3, so that part still rests on the 10-30 phone test.