Self-hosting on a Raspberry Pi cluster

And know that virtually 100% of software will run on it, rather than having to hope there's an ARM build of it (which most major FOSS software does have, but there's still that slice that doesn't)

And even if you want to experiment with multi-node HA stuff, I personally prefer to do that with VMs than wrangling actual physical machines

i have a raspberry pi3 (4gb, 50 euros) and a pi5 (8gb, 100 euros). i run a k0s cluster (for fun) w/ pihole, plex, some monitoring and homeassistant. that for me is cost effective

As to your direct question: I used some Pis for TV dashboards at work and had some random bit flips on the SD card and corrupted files for the dashboards. It may be "rare," but seems inevitable on a long enough timeline. For toy projects where you can re-image the SD card it's alright, but even for my trivial personal projects it made me uneasy.

100% to backups. I know we all put off doing it, but you'll rest a lot easier, even with personal data you don't think you care about. It's not only about a hardware failure, but even a fluke sysadmin error where you accidentally nuke something. I'd recommend getting a account for Backblaze B2, and setting up restic on each Pi to at least daily backup the data directories and stuff you care about. For your Gitlab it's a bit less risky since presumably you also have a clone of each repo on some other machine.

I love that people are building small datacenters out of Pis. I haven't done the math as far as TCO, but instead of multiple Pis for self-hosting, I have a lonely secondhand Dell Precision with an old 8th gen Intel CPU (6C/12T), 64GB of RAM, and several TB of NVMe plus some spinning rust for the long term stuff. It's just a crazy amount of horsepower. Most trusted workloads run as containers, and my other experiments can run as VMs, and I have capacity in all the right places (I need disk and RAM more than CPU). Not as exciting as building a cluster, but I have the excess capacity to spin up multiple VMs on that one machine, if I want to play with that. It can get very Inception-like, what if I'm running VMs in KubeVirt on top of Kubernetes that is running on a cluster of VMs that are ultimately on a single machine, but while delegating whatever extra /64 IPv6 prefixes Comcast gave me to each of the bottom-layer VMs so that each pod still gets a globally routable IPv6 address. Cool times for the homelab stuff, and helped me understand things like Kubernetes and IPv6 to a much greater depth.

Only time I've ever had an issue is when I manually disconnected power from a Pi 4 when I overreacted to a network outage. I was dumb and thought my Pi's DNS was screwing things up for the whole network, but it turns out that it was just Spectrum screwing things up for the entire region. Fortunately there was only one minor corrupted file, it didn't bork the SD card, and I was able to repair the setup manually.

I've never had any problem as long as I've stuck with the `sudo shutdown` command, which powers down in a controlled manner so writes don't get interrupted. But I've also never had an issue after a power outage, so I think somehow power outage shutdowns are more graceful than just yanking the power cable out.

If you're really concerned, you can always configure your Pi to use a read-only root filesystem. Combine that with a USB SSD to store your _actual_ data and you should be OK indefinitely. Or just boot from that USB SSD -- just remember to configure fstab correctly for your intended behaviour! You might be surprised to discover that fstab can delay a boot indefinitely if mounting expectations do not match reality.

I’ve had 3 SD cards fail on me in the last year. I now avoid using them as serious long term storage.

To be fair, these SD cards were exposed to fairly extreme Texas temperatures. One in a car dashcam, the other in an outdoor camera.

$6 vps probably has no storage. E.g 5tb hetzner box is more like $14 a month. Pis are absurd though for homelabbing today, prices have not fallen relative to other cheap compute. For the price of a $90 pi you can get a used mini pc with an i5 8500 with memory and storage on ebay today. Double that and you can get a new asus minipc with an 8 core 9700t. All with a case and psu. The asus minis are even designed to stack nicely as a cluster arrangement.

From my experience, being comfortable with networks and systems outside the cloud is a valuable skill that not a ton of people possess. It may be surprising to some of you, but there is an entire generation of syadmin/devops that have never touched a switch or a bare metal server.

When you put your data on a third party host, you can basically kiss it goodbye -- "trusting" that host means you have to trust every single employee in it.

For the peace of mind, I'd even pay more -- but it's significantly cheaper in the long run to self host since those 'cheap shared-resource' virtual servers are always oversold.

But to each his own.

Industry-wise Docker Swarm is somewhere between "on life support" and "shambling corpse" and has been for a while. I'd say for at least as long as since the Mirantis acquisition in 2019.

I mean in all honesty running docker swarm instead of k3s or k8s seems ... i dunno, lol, i've slowly moved away from using docker even on my single hosts i run podman now instead, i like using systemd for containers :S