• 0 Posts
  • 19 Comments
Joined 1 year ago
cake
Cake day: June 23rd, 2023

help-circle


  • K8s has a mild solution to chicken and egg situations for nodes - the nodes support ‘static manifests’ which can be pods they know how to bring up before ever connecting to the API server. So you could have your wireguard peer be brought up this way. Downside is while those static manifests show up in k8s APIs, they aren’t fully manageable since they are defined by files on disk.










  • Ceph is excellent as a distributed storage solution - but should really have 4 machines with 2 or more drives each to reach a good level of redundancy - which is a bit much for most people on this sub.

    One nice feature is it deals with heterogeneous drives well, like if you need to buy a bunch of used ones on eBay for cheap.

    Probably not a good solution for your case because of the footprint - but good to be aware of it.






  • I have 2 pi4 4GB boards and was waiting forever to get a third to run RAFT based services across.

    I gave up last year and bought 3 chinese boards at $60/ea with 2x 2.5Gb Ethernet each, emmc, and a m.2 slot - and they run at half the temp of the pi4 boards.

    I never needed the wifi/bt and form-factor the pi boards offered anyway - really no reason to stay as long as you can find software that boots on other boards.



  • Ceph (rbd,s3) on 4 poweredges.

    Nomad, Consul servers running in a 3 node raft on some ARM SBCs.

    Nomad clients on 2 poweredges and 3 arm SBCs running:

    • s3 CSI (media and large file storage is all on s3)
    • rbd CSI (all the stateful jobs have rbd block devices)
    • NATS cluster with Jetstream to enable MQTT support
    • mosquito mqtt (had to run mosquito for mqtt v5)
    • mosquito<->nats bridge (via benthos)
    • nodered (just for easy Google home integration)
    • zwavejs2mqtt
    • zigbee2mqtt
    • frigate (can only talk mqtt v5 so had to run mosquito just for this)
    • grafana
    • gotify
    • gitea
    • drone CI
    • postgresql
    • BitTorrent client
    • ceph rgw s3 gateways
    • NATS based home automation lambdas - I wrote these in go
    • Adguard home
    • traefik as main ingress
    • Prometheus
    • prom node exporter
    • jellyfin
    • jackett
    • a program I wrote that manages torznab->acquisiton->s3 lifecycle
    • a website
    • wireguard servers

    And that is just in the server room - I also have more like the 3d printer and CNC machine controllers etc.