The death and rebirth of my home server
sgt.hootr.club
sgt.hootr.club
The current setup writes to the systemd journal at least twice every minute... It's next iteration is getting a proper nvme.
Somebody set up us the bomb.
I've just scrapped a large PBX which had a whopping 8MB SD card in it. Its 486 processor booted Linux off the SD card, it wrote CDRs to the SD card, it served said CDRs up over a web interface off it, it saved its logs to the card, and all the other day-to-day chatter of a running system.
It was installed pretty much 20 years ago. The date sticker on the SD card shows that it was replaced 18 years ago, presumably as part of an upgrade.
The factors for SD card failures in raspberry pi seem to be mostly SD cards not meant for the purpose and having things like swap, atime or disk logs enabled which leads to unnecessary writes.
Meanwhile something like the pi, on most images, is writing logs to the storage CONSTANTLY. Not to mention is regularly booted.
Rebooting it was a big deal though, so that rarely happened.
It really is a high failure rate.
I have always bought the official power supply, yet of the two Pi 3s that I have connected to my TV at home, the one that runs Kodi/OSMC to serve up movies has never suffered any filesystem corruption, despite it suffering repeated power outages without warning (it's low enough to the ground that the toddler has often pulled the power cable out, and I haven't been able to find a better place to keep it). The other one, bought at the same time from the same source, is meant to be the one the kids use for programming in Snap!, a variant of MIT Scratch that doesn't have the artificial limitations that Scratch has. (Snap! functions — the exclamation mark is part of the name — can take lists as input, whereas Scratch functions can only take numbers or strings, giving it an artificial ceiling; also, Snap! functions can take functions as input and return functions, making higher-order programming possible, while Scratch deliberately does not allow that). However, that Raspberry Pi has suffered two SD card failures in the past year; it's been down more often than it's been up, and I've ended up using either my laptop or my wife's laptop for my kids' programming lessons.
More than likely due to being in a warm box...
People probably see more of a problem on raspberry pis because they write more to the memory than other systems.
Didn't even check, just took and sent. Then went back.
Only to find that the SD card in the phone was dying, and every image was corrupted from about 1/3 down...
It was an old wyse terminal, and at some point a CF adapter with an SD card was added, and a construction project buried it inside of a block wall. lol. It’s still there.
We came one day to a plant room where our rack was, to find it was in an entirely different "room", with a locked door....
Since then I’ve been careful to buy high quality cards, oversized them a bit, optimize the OS and apps with regards to writes (like no unnecessary logging), and paid attention to airflow. The software probably also evolved to fix the issues.
Let alone things like putting /var/tmp and/or /var/log in RAM, or long commit intervals.
No idea what the current situation is; Pis are too dear for the likes of me now.
relatime is a default for a while now, like decade+, you don't need to set up mount option just to get relatime
Makes it easy to replace, though.
You can buy cards that support SMART if you want to track wear, but they are expensive.
What I'm saying is, your anecdotal evidence of two pis doesn't justify saying we did something wrong with the configuration. In my experience raspberry pis still behave like "throwaway devices" but their price tags do not justify it anymore.
Rpi5 has a pcie slot if you want a cleaner setup.
I bought an old Lenovo M92 on eBay, threw in an old 500GB HDD, and it worked perfectly for the job. I had thought about just buying a newer Pi but the 14 year old Lenovo offered better specs at only $38. I promptly bought another more modern version for $100 for my desktop.
Based on benchmarks for an identically specced machine [1], a new Pi 5 edges out on single core performance and TDP but gets beat badly on multi-core and price (4-5x, even on eBay!). There also would have been the extra cost of buying a case for the Pi. I could have spent even more money on a newer mini PC that blows the Pi 5 away entirely. Obviously a Pi (even a Pi 2) is plenty capable of many tasks, I have a Pi 3 running my home assistant but it's hard to justify the extra cost over surplus thin clients that can be purchased for much less. The Pi does have its GPIO bank that's just not offered on any mini PC but I can barely remember how many times I've needed both GPIO and a whole PC-like stack to run code, an ESP8266/32 have worked fine. An external PCIe bus is cool but again, unless you're doing some really unique edge computing, not really much of a selling point.
Power consumption is a concern but they are both so low that unless you're operating off a battery, power consumption is negligible relative to everything else in your house. It would take 2 years to make up the difference in price running at full TDP (38 vs 12W). 38W is low enough that fan noise is barely a concern as well.
Size is also a concern but the difference in size is negligible unless you're trying to cram as much compute power as possible into a mini rack. A mini PC is probably the same size as your home router so just put it there.
I just really can't see the need for running the full size Pis anymore. An old mini PC does a better job for cheaper for server tasks. A Pi Zero 2 or even a Pico are much more inline with the classic Raspberry Pi ideology.
[1] https://browser.geekbench.com/v6/cpu/compare/4383152?baselin...
But other avenues exist now, including just old hardware.
This is not specific to RPis. When an SD card is subjected to "unpredictable" writes it creates potential risk of corruption. The writes are triggered by software, not the RPi hardware
There is no rule that says the RPi user must mount the root filesystem r/w on the card. It can be mounted r/w on mfs or tmpfs, for example, and the card can be removed after boot. Been doing this since 2012
Zero writes to the card, no corruption
It's true these corruption issues are "notorious" but that's due to RPi owner behaviour, not the RPI hardware
The recommended OS during this process is .... "Raspberry Pi OS" i.e. Raspbian.
When I bought RPi Model B the hardware did not come with an OS pre-installed
It booted only from SD card
It was up to me to choose which OS to run. I did not choose any of the "recommended" operating systems. I was already booting other "diskless" computers using USB sticks with root mounted on tmpfs so I wanted to use that same OS for the RPi
I never experienced SD card corruption with RPi hardware
However I have experienced SD card corruption with other SBCs that had an OS pre-installed, an OS not chosen by me
By choosing to run a "recommended" OS on the RPi perhaps I would experience SD card corruption
But if that happened I would not attriubute it to the RPi. I would blame myself. And I would make different choices going forward
To use your Raspberry Pi, you must first install (flash) an operating system (OS) onto your storage device, making it a boot device. By default, Raspberry Pi devices check for an OS on any SD card inserted into the SD card slot. Some Raspberry Pi models allow you to boot an OS from other storage devices, such as USB drives, storage connected through a HAT, and network storage. The following instructions assume that you're using a microSD card.
We recommend installing an OS to your storage device using Raspberry Pi Imager on another computer. For instructions, see ~Install using Imager. If you don't have access to another computer for writing an OS image to the storage device, you might be able to install an operating system directly on your Raspberry Pi from the internet. For instructions, see ~Install over the network."
https://web.archive.org/web/20260715154847/https://www.raspb...
What is the difference between a "recommendation" and a "default"
Here, we can see that this is a recommendation not a default
The owner must make a choice
The company has not pre-selected a default choice
It has only provided recommendations
Contrast "defaults" with "recommendations"
The former is opt-out; a choice has already been pre-selected by the company
The later is opt-in; no choice has been pre-selected, a choice must be made by the owner
The recommended software, derived from Debian,^1 and the default configuration of that software is what causes the SD card corruption, not the RPi development board
1. If I am not mistaken the OS recommendations have changed since the RPi was released, e.g., the company was originally recommending RISC OS. Challenge me on this and I'll find the reference
True SD cards are less than ideal.
But also I suspect half the problem specifically with Pi and SD cards is that people use cheap-ass SD cards and maybe ones they found in the bottom of a drawer that may or may not have previously been used in another device (e.g. camera).
I suspect if people bought industrial SD cards instead of consumer-grade junk they might get a better lifespan out of them.
That’ll keep the rasp online for essentially forever cause optane isn’t going to wear out
I agree that its a good, long-lasting fix, but when was the last time you checked the pricing on those?
If you can even find any, they certainly wont be cheap nowadays!
Though I'd agree that if you got none the NUC is the better solution as if you add an ssd and power supply to the PI it becomes as if not more expensive for less power
Don't get me wrong: I've got like seven Pis, 4 NUCs (but only one "recent" N100 one) and even though two of the Pis are up 24/7 (one runs the unbound DNS and the other, well, serves as a router/firewall to isolate my main PC from the rest of the machines) my server, as in the machine that has disks in RAID (ZFS), runs VMs, containers (including Immich and Jellyfin), etc. is a Xeon with ECC RAM.
I had no idea a Pi 4 was that beefy.
Now of course booting off a microSD card: yuk (I know, I know, there are read-only trickeries that can be achieved).
> The important stuff mostly lived in external HDDs and the configuration lived almost entirely in my NixOS configuration.
Yeah NixOS or what-have-you: if you have server, I'd recommend you have a plan B for when it fails. My little infra at home is mostly automated: it's all scripts provisioning the VMs, setting up cloudinit for the first boot, etc.
As long as I've got backups of my config/scripts and of my data, setting up a new server really doesn't take long (heck I did it not long ago, mostly to be sure the "death and rebirth" procedure of my own server was sound).
I personally prefer my 10+ years old Xeon with ECC as a server than my N100 NUC but then the Xeon consumes more energy... YMMV.
Been 12 years or so now. Went back to their home a few years ago and everything but the opensubtitles integration works. I was surprised. I could never recreate it now. Even has one of these flirc setups so the remote works.
India has unreliable power too so the SD card death was inevitable. I made a few more images and called it a day.
Thankfully, modern Rockchip SBCs tend to come with NVMe slots.
A few days ago the HDD had its 100,000 Power_On_Hours anniversary. :) According to SMART this is way past the expected/ designed life time of the disk (the normalised value reports "1" since a few years when the disk hit about 42,000 Power_On_Hours)
So you never know. It either works or it doesn't. You have backups for the worst case.
With 21,647,615 MiB (~20.64 TiB) written. Reporting as 99 (down from initial 100?) in the normalised value, but the normalised value hasn't changed since I first checked in 2015, so maybe it doesn't work? The wear leveling is at 180 (down from initial 200?), and the threshold at 100, so still ~80 to go until expected EOL. I'm losing between 1 and 2 points per year at the current rate.
I'm also hoping for another 10 years at least. *knock-on-wood*
Uptime is fine, until you restart. And poof
And especially for home use, where nien 9s is more realistic than nine 9s, you're better off having a solid backup strategy and a source of spare parts.
My main "tank" server is 16 years old this year, and has blown out one motherboard - it was a bit annoying to be "offline" for a few days as a replacement wandered in from eBay, but it worked. If it happened today I might import the zpool into another device temporarily - or permanently, who knows? This R510 probably takes more power than a datacenter.
Hardware failures tend to follow a bathtub curve: there are more frequent failures early on due to manufacturing defects, then they become rare until you reach the end of the natural lifespan of the equipment. That is shorter for some components (spinning rust HDDs, fans, CMOS batteries, power supplies in noisy environments), but extremely long for others.
Some linux distribution will regularly check SMART data from disk and warn you if some threshold has passed (like number of corrected errors, spin up/down, total hours etc).
That said, if we talk about consumer hardware in a home lab or the like, of the disks that I bought in the past 15 years, only one broke down, and it did fairly quickly so my suggestion is to use redundant storage, make backup and change the disk when it breaks down. (Enterprise disk and usage is another can of worms of course).
You can get unlucky and have components fail quickly, but usually if they last, they'll last for a long time.
But trying to run a server from an SD card like in the article, seems silly. False economy, just get a good SSD.
For drive health monitoring there is SMART, but IME while it does monitor drive usage and lifetime, it rarely predicts catastrophic failure since that happens suddenly. Both drives that failed in this system went from healthy (according to SMART) to unresponsive overnight.
Do most people actually use them? Pretty rarely. S.M.A.R.T is just trash anyway.
But you do need to properly design your systems to limit the amount of "use". No un-needed writes to disk etc. Proper ambient detection, no point trying to rush to shut down servers when the AC pops it, and you just so happen to walk into a server room.
But this is the kind of experience that makes technical people fall in love with self-hosting, but also keeps the majority of the population away.
Most of the people don't want the hassle of managing a self-hosting box.
I think a middle ground option might where someone hosts it for you, but you can also move to your own hardware at any time might be best
swap is used when you run out of ram. using a ramdisk for swap ... You see how that doesnt make sense?
This is not correct. Swap is not a RAM overflow.
> Swap is not generally about getting emergency memory, it's about making memory reclamation egalitarian and efficient. In fact, using it as "emergency memory" is generally actively harmful.
You can compress the swap, so you can squezze a bit more by using CPU in that tiny amount of read you do.
Use a clanker. Nothing has brought back so much joy into homelabbing for me as having LLMs handle the boring stuff.
I launch it in opencode on the system or elsewhere to ssh in (I think giving it KVM is an overkill but an option nonetheless) and tell it to fix things. Not just ask questions and generate configs. I have backups, let it rip. They have gotten shockingly good at it even when they write bash wrappers just to catch some logs.
Back in the day I ran s××× coin miners off flash drives without any apparent problems.
Teslas though... ;)
every other component is cheap enough but not Ram.
I bought all the stuff (new case, networking card and a Jonsbro N4 to repurpose some intel i7-9700 era processor and motherboard, but didn't get it setup yet, as I was starting to rethink the power consumption....
Also looking to use this rebuild as an opportunity to learn Nix.
Let me know if you all have some good article references, to point me at. I've been doing some research, but could always do more.
I may end up putting a low power NUC in the case, or something. I don't know...
Yes! It runs completely fine on a Pi, except for the machine learning service. The good news is that you can run that service on a separate computer with a GPU and point Immich to it. Immich will use it when it's available and I think you can configure it to run all the queued ML jobs at a certain time when you expect the computer running the ML service to be on.
I now use a small SSD connected via USB3 and a slightly modified version of this : https://framps.github.io/raspiBackupDoc/introduction.html
...to take snapshots to my NAS periodically.
I keep a similar small SSD in the drawer ready for a restore and swap if the main system drive fails.
And every few years reinstall the OS to free up unwritten cells.
I assume it’s a bad capacitor or something. I tried changing out the power adapter, but that didn’t help.
I don’t see myself ever buying another Beelink.
I run a Proxmox server on my Dell Optiplex SFF 7070. With the SFF size you can add two low-profile PCIe cards, if desired. In my case I added a 10Gbit NIC.
The reason I would go with a used Dell / Lenovo / HP is that these tend to be well-built machines that run for years and years 24/7 in offices and proven to work well.