I admire the courage to store data on refurbished Seagate hard drives. I prefer SSD storage with some backups using cloud cold storage, because I’m not the one replacing the failing hard drives.
But even the cheapest high capacity SSD deals are still a lot more expensive than hard drive array.
I’ll continue replacing failing hard drives for a few more years. For me that has meant zero replacements over a decade, though I planned for a 5% annual failure rate and have a spare drive in the case ready to go. I could replace a failed drive from the array in the time takes to shut down, swap a cable to the spare drive, and boot up again.
SSDs also need to be examined for power loss protection. The results with consumer drives are mixed and it’s hard to find good info about how common drives behave. Getting enterprise grade drives with guaranteed PLP from large on-onboard capacitors is ideal, but those are expensive. Spinning hard drives have the benefit of using their rotational inertia to power the drive long enough to finish outstanding writes.
Not to say you shouldn't backup your data, but personally I wouldn't be to affected if one of my personal drives errored out, especially if they contained unused personal files from 10+ years ago (legal/tax/financials are another matter).
Mainly I don't want to lose anything that took work to make or get. Personal photos, videos, source code, documents, and correspondence are the highest priority.
https://www.servethehome.com/we-bought-1347-used-data-center...
I bought over 200 over the last year, and the average wear level was 96%, and 95% had a wear above 88%.
After going deep on the spec sheets and realizing that all but the best consumer drives have miserably low DWPD numbers I switched to enterprise (U.2 style) two years ago. I slam them with logs, metrics data, backups, frequent writes and data transfers, and have had 0 failures.
Software solutions like Windows Storage Spaces, ZFS, XFS, unRAID, etc. etc are "just better" than traditional RAID.
Yes, focus on 2x parity drive solutions, such as ZFS's "raidz2", or other such "equivalent to RAID6" systems. But just focus on software solutions that more easily allow you to move hard drives around without tying them to motherboard-slots or other such hardware issues.
I like btrfs for this purpose since it's extremely easy to setup over cli, but any of the other options mentioned will work.
RAID0/1/10 has been stable for a while.
"This release introduces the [Btrfs] RAID stripe tree, a new tree for logical file extent mapping where the physical mapping may not match on multiple devices. This is now used in zoned mode to implement RAID0/RAID1* profiles, but can be used in non-zoned mode as well. The support for RAID56 is in development and will eventually fix the problems with the current implementation."
I've not kept with more recent releases but there has been progress on the issue
> btrfs is quite infamous for eating your data.
This is the reason for the slogan on the bcachefs website:
"The COW filesystem for Linux that won't eat your data".
After over a decade of in-kernel development, Btrfs still can't either give an accurate answer to `df -h`, or repair a damaged volume.
Because it can't tell a program how much space is free, it's trivially easy to fill a volume. In my personal experience, writing to a full volume corrupts it irretrievably 100% of the time, and then it cannot be repaired.
IMHO this is entirely unacceptable in an allegedly enterprise-ready filesystem.
The fact that its RAID is even more unstable merely seals the deal.
> In my personal experience, writing to a full volume corrupts it irretrievably 100% of the time, and then it cannot be repaired.
While I get the frustration, I think you could have probably resolved both of them by reading the manual. Btrfs separates metadata & regular data, meaning if you create a lot of small files your filesystem may be 'full' while still having data available; `btrfs f df -h <path>` would give you the break down. Since everything is journaled & CoW it will disallow most actions to prevent actual damage. If you run into this you can recover by adding an additional disk for metadata (can just be a loopback image), rebalancing, and then taking steps to resolve the root cause, finally removing the additional disk.
May seem daunting but it's actually only about 6 commands.
What that means is I wrote the manual.
Now, disclaimer, not that manual: I did not work on filesystems or Btrfs, not at all. (I worked on SUSE's now-axed-because-of-Rancher container distro CaaSP, and on SLE's support for persistent memory, and lots of other stuff that I've now forgotten because it was 4 whole years and it was very nearly 4 years ago.)
I am however one of the many people who have contributed to SUSE's excellent documentation, and while I didn't write the stuff about filesystems, it is an error to assume that I don't know anything about this. I really do. I had meetings with senior SUSE people where I attempted to discuss the critical weaknesses of Btrfs, and my points were pooh-poohed.
Some of them still stalk me on social media and regularly attack me, my skills, my knowledge, and my reputation. I block them where I can. Part of the price of being online and using one's real name. I get big famous people shouting that I am wrong sometimes. It happens. Rare indeed is the person who can refute me and falsify my claims. (Hell, rare enough is the person who knows the difference between "rebut" and "refute".)
So, no, while I accept that there may be workarounds that a smart human may be able to do, I strongly suspect that these things are accessible to software, to tools such as Zypper and Snapper.
In my repeated direct personal experience, using openSUSE Leap and openSUSE Tumbleweed, routine software upgrades can fill up the root filesystem. I presume this is because the packaging tools can't get accurate values for free space, probably because Btrfs can't accurately account for space used or about to be used by snapshots, and a corrupt Btrfs root filesystem can't be turned back into a valid consistent one using the automated tools provided.
Which is why both SUSE's and Btrfs's own docs say "do not use the repair tools unless you are instructed to by an expert."
For my perspective, my experience with btrfs has been flawless through 11 machines and at least 3 major releases on each without any maintenance but it could just be I'm not hitting the worst case (only use snapshots on 2 machines, raid on 3). And I've only used btrfs since fairly recently (~4 years now). I've had to recover one drive of a friend using the method I outlined before as he filled the entire drive with media. For me the trade-off of a few rough edges but more functionality & flexibility than other filesystems is worth it.
For your update issue, I think you're mostly correct; the package manager likely assumes the filesystem is not snapshotted (i.e. it will reclaim disk space), while btrfs with snapshots/CoW will use the entire size of written files unless it's in the same snapshot.
It's a balance. All of life is a balance.
I worked at Red Hat very briefly, and for SUSE for longer than ever before. Both were good workplaces with a good atmosphere: RH is one of the friendliest places ever, and I'm still friends with former colleagues from over a decade ago.
OTOH, installing Fedora 14 was like having a bucket of cold water to the face. I used and reviewed Red Hat Linux in the 1990s and it was a massive PITA. It had no automatic dependency resolution, so complex software installation (e.g. going from KDE 1.x to KDE 2.x) was a huge task involving manually installing hundreds of dependencies.
RH would not bundle KDE (because Qt was not 100% FOSS) -- which is also why Mandrake was founded -- and so the GUIs on RHL were poor.
I switched to SUSE. Good package management, good GUIs, good system-management tools. (YaST was way better than RH's inadequate `linuxconf`.)
RH "fixed" this by... removing Linuxconf.
Trying Fedora a decade later and it was just as bad. All the external bits improved because upstream improved. GNOME was still a mess. KDE had got much more bloated. Xfce was better than ever.
But the RH in-house bits, while having nice visual design, were functionally terrible. The installer was an embarassment.
Over a decade of work since I reviewed RHL 9 and it was worse than ever.
A few years later, go to work at SUSE, and hey, openSUSE was lovely. All the good bits still there and improved. Yeah, a bit bigger and slower and clunkier than Ubuntu.
But there's always a downside.
An older team so less party atmosphere. Fewer "team building" sessions in the pub.
And while I was away, SUSE switched from ReiserFS to Btrfs, and as usual, SUSE of old being fond of experimental bleeding-edge filesystems, it's half-implemented and doesn't work right.
Snapper doesn't prune snapshots thoroughly enough. It fills your disk and because of unimplemented or non-working features it can't tell when this will happen.
Official SUSE answer: give it lots of space. Here, our FS falls over unrepairably when full, so give it all your space so it won't fill up! And you can't repair it so take lots of backups!
Every distro has downsides. Every filesystem has downsides but they are much less obvious.
Btrfs made my openSUSE boxes collapse and crash 2-3 times a year for 4 years. That is intolerable. I put up with that kind of crashy junk in 1997 or so but not 20 years later.
RAID does not mean or imply hardware RAID controllers, which you seem to incorrectly assume.
Software RAID is still 100% RAID.
------
The best advice I can give is to use a real solution like ZFS, Storage Spaces and the like.
It's not sufficient to say 'Use RAID' because within the Venn Diagram of things falling under RAID is a whole bunch of shit solutions and awful experiences.
It's still enabled in the firmware of some vendors' laptops -- ones deep in Microsoft's pockets, like Dell, who personally I would not touch unless the kit were free, but gullible IT managers buy the things.
My personal suspicion is that it's an anti-Linux measure. It's hard to convert such a machine to AHCI mode without reformatting unless you have more clue than the sort of person who buys Dell kit.
In real life it's easy: set Windows to start in Safe Mode, reboot, go into the firmware, change RAID mode to AHCI, reboot, exit Safe Mode.
Result, Windows detects a new disk controller and boots normally, and now, all you need to do is disable Bitlocker and you can dual-boot happily.
However that's more depth of knowledge than I've met in a Windows techie in a decade, too.
Redundancy rather than individual reliability.
Storing 18TB (let alone with raid) on SSDs is something only those earning Silicon Valley tech wages can afford.
In essence, what we together are saying is that people with super-sensitive sleep that are also easily upset, and that don't have ultra-high salaries, cannot really afford 18 TB of data (even though they can afford an HDD), and that's true.
And you should use some form of redundancy/backups anyway. It's also a good idea to not use all disks from the same batch to avoid correlated failures.
1: https://datablocks.dev/blogs/news/white-label-vs-recertified...
Quote from Toshiba's paper on this. [1]
Hard disk drives for enterprise server and storage usage (Enterprise Performance and Enterprise Capacity Drives) have MTTF of up to 2 million hours, at 5 years warranty, 24/7 operation. Operational temperature range is limited, as the temperature in datacenters is carefully controlled. These drives are rated for a workload of 550TB/year, which translates into a continuous data transfer rate of 17.5 Mbyte/s[3]. In contrast, desktop HDDs are designed for lower workloads and are not rated or qualified for 24/7 continuous operation.
From Synology
With support for 550 TB/year workloads1 and rated for a 2.5 million hours mean time to failure (MTTF), HAS5300 SAS drives are built to deliver consistent and class-leading performance in the most intense environments. Persistent write cache technology further helps ensure data integrity for your mission-critical applications.
[1] https://toshiba.semicon-storage.com/content/dam/toshiba-ss-v...
[2] https://www.synology.com/en-us/company/news/article/HAS5300/...
Max operating range is ~60C for spinning disks and ~70C for SSD. Optimal is <40-45C. The larger agents facilties afaik tend to run as hot as they can.
It doesn't apply for the single drive, only for a large number of drives. E.g. if you have 100000 drives (2.4 million hours MTTF) in a server building with the required environmental conditions and maximum workload, be prepared to replace a drive once a day in average.
If you're buying them from the second hand market, you don't likely get the warranty (and is likely why they're on the second hand market)