Are you backing up files from the FS or sre you backing up the snapshots? I had a corrupted journal from a power outage that borked my install. Could not get to the snapshots on boot. Booted into a live disk and recovered the snapshot that way. Would’ve taken hours to restore from a standard backup, however it was minutes restoring the snapshot.

If you’re not backing up BTRFS snapshots and just backing up files you’re better off just using ext4.

https://github.com/digint/btrbk

Do you rely on snapshotting and journaling? If so backup your snapshots.

Considering it’s tailscale, one may want to have the service fail though as tailscale is sometimes not used for convenience but security concerns instead.

Delay the start of your containers with the tailscale dependency. Are you using required or depends_on in your docker-compose.yml

https://hatchjs.com/docker-compose-conditionally-start-service/

If you’re using kubernetes you can make the requirements at the pod level

https://stackoverflow.com/questions/69423932/container-initialization-order-in-pod-on-k8s

Edit: If using docker-compose.yml you can set a condition on a healthcheck

You can also specify a condition that must be met before the service is started. For example, the following configuration will start the web service only if the db service is healthy:

version: ‘3.7’
services:
web:
image: nginx
depends_on:
– db
condition: service_healthy

The service_healthy condition checks the health of the db service using the docker-compose ps command. If the db service is healthy, the web service will be started.

In my humble opinion the point of self hosting an LLM is so that the data doesn’t leave your LAN.

Yes, but what does the BLOB do?

It’s as complex as you make it, is linux native, is scriptable, doesn’t use YAML, is native to the OS, and is free as in beer. Just like SNMP. however they’ll also get logs at a central server they can drill into if needed.

Which I believe fulfills the requirements of OPs post.

Sidenote, self-hosting is absolutely overkill just as a theory and process. I often read responses to suggestions as this or that is overkill, or complicated, or non-trivial effort.

The self hosting community is a broad spectrum of users , from those with home labs on an old dying laptop to those with a full rack setup. People have different needs and interests. Some are learning infra and devops for work or to get into a new job. Some are privacy minded. Some are trying to get the most bang for their buck. Some just want to pay for a cloud hosted solution. Some just want an automated home. Some run a home business.

Edit: to the point of your valid and helpful SNMP post, most syslog servers also will ingest and report on SNMP traffic as well. The container I linked does exactly that. If they find they want to automate processes in the future they can also trigger on the syslog stream as well. But that complexity is only there if they want it. Otherwise it’s just a stream they can parse and trigger an alert, just like SNMP. So OP could have an extensible solution if they want to expand. Also Grafana/Prometheus will take in syslog natively with a couple standard YAML configs if they choose that they want to look at that solution again in the future.

/Rant

Edit: found better resources

https://linuxhandbook.com/syslog-guide/

https://github.com/linuxserver/docker-syslog-ng

That should be a good place to start. Syslog will do what you want.

Proxmox also can run on ZFS, has support to run containers, and can also manage backups.