Anything exposed to the internet will be found by the scanners. Moving ssh off of port 22 doesn’t do anything except make it less convenient for you to use. The scanners will find it, and when they do, they will try to log in.

(It’s actually pretty easy to write a little script to listen on port 20 (telnet) and collect the default login creds that the worms so kindly share)

The thing that protects you is strong authentication. Turn off password auth entirely, and generate a long keypair. Disable root login entirely.

Most self-hosted software is built by hobbyists with some goal, and rock solid authentication is generally not that goal. You should, if you can, put most things behind some reverse-proxy with a strong auth layer, like Teleport.

You will get lots of advice to hide things behind a vpn. A vpn provides centralized strong authentication. It’s a good idea, but decreases accessibility (which is part of security) - so there’s a value judgement here between the strength of a vpn and your accessibility goals.

Some of my services (ssh, wg, nginx) are open to the internet. Some are behind a reverse proxy. Some require a vpn connection, even within my own house. It depends on who it’s for - just me, technical friends, the world, or my technically-challenged parents trying to type something with a roku remote.

After strong auth, you want to think about software vulnerabilities - and you don’t have to think much, because there’s only one answer: keep your stuff up to date.

All of the above covers the P in PICERL (pick-uh-rel) for Prepare. I stands for Identify, and this is tricky. In an ideal world, you get a real-time notification (on your phone if possible) when any of these things happen:

• Any successful ssh login
• Any successful root login
• If a port starts listening that you didn’t expect
• If the system watching for these things goes down (have two systems that watch each other)

That list could be much longer, but that’s a good start.

After Identification, there’s Contain + Eradicate. In a homelab context, that’s probably a fresh re-install of the OS. Attacker persistence mechanisms are insane - once they’re in, they’re in. Reformat the disk.

R is for recover or remediate depending on who you ask. If you reformatted your disks, it stands for “rebuild”. Combine this with L (lessons learned) to rebuild differently than before.

To close out this essay though, I want to reiterate Strong Auth. If you’ve got strong auth and keep things up to date, a breach should never happen. A lot of people work very hard every day to keep the strong auth strong ;)

I strongly recommend the NAT loopback route over attempting split-horizon dns.

At what point does a collection of microservices become a monolith that uses http instead of a bus 🤔

Tailscale might be the best bet at this point. It will manage the wireguard mesh for you, and use nat holepunching for handshaking instead of needing listening ports.