Agree, I've also been running a number of PIs and when they broke, it was because of failing SD cards.
I found pibenchmark to be a good source of info --> https://pibenchmarks.com/
Definitely compare SD cards before buying.
> I guess it all boils down to good SD cards and stable power supplies.
I don't run a completely storage oblivious setup, I use f2fs, which has much better write patterns than ext4, and I disable stats collection in postgresql, which causes constant churn, but logging, and other stuff has negligible effects and I don't care doing anything special about it, and I leave it default OS configuration.
I have probably 4-5 uSD card failures over 60 year total runtime of my 15 SBCs that I run 24/7. So that's 1 failure per 12 years. Nothing that can't be dealt with using backups. (I didn't even need them that much, yet, because all uSD cards so far failed by turning read only, which for f2fs means you'll get the last consistent snapshot on the card, that you can just copy to a new card and continue. 10-15 minutes of recovery time.
All that complexity and limitations that people talk about seem way overkill for a home setup.
And I think other reason why I don't get many uSD card failures is because I run most cards in HS25 mode. Not in SDR104 or whatnot. 3-4x the frequency really causes the cards to heat up a LOT during activity. Can't be great for the flash chips. 2 of those failures were in SDR104 enabled hosts. Copying data to uSD card using SDR104 capable USB3.0 adapter really makes the card scarry hot in just a few seconds.
While I definitely agree that for a home setup it is overkill, in my dayjob I work on industrial embedded gadgets, sold 10-100k pcs/design, expected to run without reboot for years, sometimes for decades. And most of the weird issues end up usually on my desk for investigation. While I admit that this might not have been completely obvious, but when I referred to the sample size, I was referring to such numbers, taken from this experience.
Context of the article is "I use Raspberry Pis around my home as everything". I don't really care about industrial anything in that context, nor 10k+ unit scale. And nobody sane does/should. At home/hobby I'm optimizing for very different things.
It's interesting if someone adds to the discussion that things predictably fail at 10k+ scale, but completely irrelevant. It's like people who read backblaze stats to go buy 2 disks from the same batch to put in raid1. That those disk mdoels fail at 2x lower rate at scale than other models is completely irrelevant to their data safety.
I’ve switched to SSDs for most of them now. It’s too much of a dice roll otherwise
In ~2022 I started to get random errors, then found out the SD card was failing. I never took the time to fix it, it had lots of little things locally I don’t feel like doing again.
It’s just so tempting to quickly ssh into the machine to hack something around, then you forget about it b cause it ~works.
But a rpi4 can run containers, so that may also be an alternative.
I suspect ansible may be a better solution for when you're using it to build machines from scratch very frequently, and you'd have more of a chance to maintain and ensure they're working over time vs once every couple of years.
The other annoying thing with ansible is that while it may install the software, it can't log me in. So still having to manually sign in to syncthing/tailscale etc even after the software is installed.