My Homelab Got Hacked - A Postmortem
phunky.cafe
A postmortem on the hack of my self-hosted Forgejo instance and a reminder to patch your software.

Welp. My Forgejo instance got popped with an RCE two days ago by CVE-2026-60004. Luckily, I noticed the following morning and had the day free to figure out what happened. Let’s dive in!

As a homelab enthusiast, I found this a very interesting post. Here are my take aways from the post that I’m implementing myself:

  • Miner detection. I’ve updated monitoring rules to now watch the CPU on my hosts. If the same thing happened to me I would not have been alerted at all as I’m doing simple up / down monitoring. Fixed.
  • Access logging. I turned on access logging for my homelab Caddy instances.
  • Log retention. I have increased the amount and retention of my logging. The hope is this will help me reconstruct what happened after a breach.
  • Logs offsite. The VPS access logs now ride along with the normal backup process, which runs hourly. The homelab side still only gets caught by the weekly VM backup, so that’s next.
  • Closed an open signup. My webtrees instance (genealogy) had self registration enabled, which is the same door this guy got hit through. Oops. Fixed.
  • Built a tool. log-inventory.sh, so “could I actually reconstruct what happened” is a command I run instead of a thing I assume.
  • gaylord_fartmaster@lemmy.worldEnglish
    1033·
    1 month ago

    I don’t think I will ever be convinced to leave anything on my home network open to the internet no matter how convenient it is.

    • lemmyvore@feddit.nlEnglish
      392·
      1 month ago

      It’s fairly safe as long as you add a strong enough form of access control. For example if you put it behind a VPN, or a SSH tunnel, or require mTLS. Even a key in a custom HTTP header or Basic HTTP auth can be good enough if the key is strong enough.

      You can further decrease the probability of drive-by bots reaching a publicly exposed service by merely scanning IPs and ports if you use a reverse proxy and hide your service FQDNs and IP.

      You can do this by using TLS certs on wildcard domains rather than explicit domains, using explicit CNAMEs for the service subdomains rather than a wildcard domain, and keeping the A/AAAA records on an obfuscated subdomain rather than the base domain. If the bots can’t figure out a FQDN they’re not getting past the reverse proxy even if they find the IP and port.

      This is obfuscation not real security but it cuts down tremendously on bot hits.

        • frongt@lemmy.zipEnglish
          52·
          1 month ago

          The VPN service is open to the Internet. But it’s only one service, versus however many things you are running behind it.

      • hietsu@sopuli.xyzEnglish
        9·
        1 month ago

        I have knockd listening on one very high port number, and only after accessing that reverse proxy whitelists the source ip. Sure some scanners might do a full port scan and try http only after that, but FQDN is required too. Plus I have geo ip list blocking all but my country. Oh and Crowdsec. So far so good…

    • leds@feddit.dkEnglish
      11·
      1 month ago

      I gave up , don’t have the time to maintain it anymore. Changed DNS to make my domain point to local lan IP so everything still works but only at home.

      • village604@adultswim.fanEnglish
        3·
        1 month ago

        I wonder if the person who downvoted you thought you pointed your public DNS record to a 192.168 network, instead of making a change to your local DNS server like I’m assuming you did.

        • jet@hackertalks.comEnglish
          42·
          1 month ago

          If you want https it’s gotta be a external DNS record anyway, might as well point to internal ips

          Depending on how paranoid your setup is not everything will use the DHCP provided DNS servers

          • KubeRoot@discuss.tchncs.deEnglish
            7·
            1 month ago

            If you want https it’s gotta be a external DNS record anyway

            Not sure what you mean by that, but off the top of my head, you can get certificates via challenges that prove DNS control (rather than checking if the DNS points to your server), and you can get wildcard certificates so you don’t even have to expose the existence of subdomains.

            And that’s ignoring the option of using your own CA, which only really works with your own devices, but for local access might be viable.

            • MadPsyentist@lemmy.nzEnglish
              1·
              1 day ago

              So what you do is you have an external DNS that sharing your URL. You have the DNS point “myrealcoolurl.net” at 192.168.1.123. Now when anyone goes to “myrealcoolurl.net” the dns points them at 192.168.1.123 which worse case is their own router, if they are on your network though they get pointed at your server and are loving all your home hosted applications.

              Now you can get a wildcard TLS cert issued for “myrealcoolyrl.net” and apps in your home network can use https. Anyone outside your network still gets whatever 192.168.1.123 means for them in their home network

              • KubeRoot@discuss.tchncs.deEnglish
                2·
                1 day ago

                I’m a bit confused by what you’re saying, because to my knowledge the DNS entry you’re mentioning won’t help with getting a certificate at all. Challenges will typically require either having the DNS entry point to a publicly reachable website (doesn’t get you wildcard) or proving the ability to modify records for the domain by adding something like a TXT record on the fly, which shouldn’t require A/AAAA records.

                So you should be able to do the second type of challenge without a global A record to get a wildcard certificate, and use local DNS for the A record?

                • MadPsyentist@lemmy.nzEnglish
                  0·
                  1 day ago

                  The challenge dosent require your site to be public only the DNS to be public. So your second description works.

                  Your serve talks to LetsEncrypt. LE gives your server a challenge. Your server adds the challenge record to your public dns. LE confirms the challenge and issues a dns-01 wildcard certificate to your server. LE never had to vist your site to confirm the challenge.

                  See for yourself and, from a terminal of your choosing, run

                  nslookup madpsy.nz

                  You should get back my local ip which doesnt work for anyone outside of my local network.

                  • KubeRoot@discuss.tchncs.deEnglish
                    2·
                    1 day ago

                    The point is, the challenge doesn’t require a public DNS A record pointing to your IP, it’s not involved in issuing a wildcard certificate. It requires you to control DNS records for the domain to issue a temporary TXT or something record to prove it, one which you can remove afterwards. This was my point in the original comment.

                    It’s A and AAAA records that are used when you open a website in a browser to map the domain to an IP address, the ones I’m saying don’t need to be public.

                    The challenge dosent require your site to be public only the DNS to be public.

                    And just to be clear, that was describing a different kind of challenge, the simpler one which most people probably use, and which does actually involve A records.