When that's said, we had routine XFS losses on SGI boxes. That was a very well known scenario: Write constantly to a one-page text file, say, every few seconds, then power cycle the machine. The file would be empty afterwards. This doesn't happen on Linux, I vaguely recall discussing this with someone some years ago (maybe here on HN) and something was changed at some point, maybe when SGI migrated XFS to Linux, or shortly after.
XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over.
For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that if the machine had a crash or sudden power loss the hardware raid would live on long enough to finish its writes. SGI had tight control over the type of hardware you could use and it was usually good quality stuff.
In the land of commodity PC-based servers this isn't often how it worked. Instead you just had regular IDE or SATA hard drives. And those drives lied.
On cheap hardware the firmware would report back it had finished writes when in fact it didn't because it made it seem faster in benchmarks. And consumers/enterprise types looking to save money with Linux mostly bought whatever is the cheapest and fastest looking on benchmarks.
So that if there was a hardware failure or sudden power loss there would could be several megs of writes that were still in flight when the file system thought they were safely written to disk.
That meant there was a distinct chance of dataloss when it came to using Linux and XFS early on.
I experienced problems like that in early 2000s era Linux XFS.
This was always a big benefit to sticking with Ext4. It is kinda dumb luck that Ext4 is as fast as it is when it came to hosting databases, but the real reason to use it is because it had a lot of robust recovery tools. It was designed from the ground up with the assumption that you were using the cheapest crappiest hardware you can buy (Personal PCs).
However modern XFS is a completely different beast. It has been rewritten extensively and improved massively over what was originally ported over from SGI.
It is different enough that a guy's experience with it from 2005 or 2010 isn't really meaningful.
I have zero real technical knowledge on file systems except as a end-user, but from what I understand FreeBSD uses UFS that uses a "WAL" or "write ahead log".. where it records writes it is going to do before it does it. I think this is a simpler but more robust solution then the sort of journalling that XFS or Ext4 uses. The trade off is lower performance.
As far as ZFS vs Btrfs... I really like to avoid Btrfs as much as possible. A number of distros use it by default (OpenSUSE, Fedora, etc), but I just format everything as a single partition as Ext4 or XFS on personal stuff. I use it on my personal file server, but it really simple setup with UPS. I don't use ZFS, but I strongly suspect that btrfs simply failed to rise to its level.
One of the reasons Linux persists despite not having something up to the level of ZFS is that most of ZFS features are redundant to larger enterprise customers.
They typically use expensive SAN or more advanced NAS that has proprietary storage solutions that provide ZFS-like features long before ZFS was a thing. So throwing something as complicated as ZFS on top of that really provides no benefit.
Or they use one of Linux clustered file system solutions, of which there is a wide selection.
I don't know about it's multi-disk story (I do use ZFS for that personally), but for single disk options it is great. You get so many of the ZFS benefits (snapshots, rollback, easily create and delete volumes, etc) with MUCH lower memory usage (at least in my own experiments to try this out).
[0] https://lwn.net/Articles/824855/ [1] https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
Edit: it's less grey now.
The cause? Filling up the filesystem. Why? Because of OS snapshots.
(Aside: why can they fill it? Because it doesn't give a straight answer to `df -h`. Why not? Because of snapshots.)
https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html
(Look for "GlobalReserve")
It happened to me repeatedly on both openSUSE Leap and openSUSE Tumbleweed during the 4 years I worked for SUSE: 2017-2021.
The `df` command doesn't work: it does not give reliable info. That alone disqualifies this FS for me.
The `fsck` equivalent does not work: every time I have tried, it corrupts volumes into unreadability.
Those 2 things are hard requirements for me.
I raised this internally as significant issues. They were dismissed.
Yeah, and when I was there machines would run out of disk space at 50% usage and it took months to figure out why. In the mean time, they'd just reimage the machine and hope. I don't recall any issues with data loss, but it didn't have the air of reliability.
But my team was weird at FB, our uptimes of 45 days were way above the average, and we ran into all sorts of things because we operated outside the norm.
ZFS is only reliable where it was battled-tested: on Solaris. ZoL has been infinitely tinkered with and smashed up that it's nothing like running a Thumper as a NAS.
XFS + mdadm on Linux is, without a doubt, far more reliable than ZoL. Ask me how I know. I have the scars to prove it.
What specific problems are there with the ZFS implementation on FreeBSD? You claim it is not battle tested, I find it to be rock solid..
Those two things are not mutually exclusive.
I also owned a Thumper and Thor running Solaris in 2009. Much prefer Linux and the hardware solutions today.
This is much different to the consumer usecase where computers are pets and not cattle. A failed filesystem the night before you need to turn in your thesis may have a much larger impact on your life.
Another thing to consider is that Facebook runs btrfs on enterprise hardware (including SSDs with battery backups) which is going to be much more reliable than some chromebook which lives in the bottom of your backpack that you bring on transit every day.
Finally, I will say that the copy on write features of btrfs can result is some wildly different behaviour based upon how you use it. You can get into some very bad pathological cases with write amplification, and if you run btrfs on top of LUKS it can nearly be impossible to figure out why your disk is being pegged despite very little throughput at the VFS layer.
I experienced issues with write amplification within the past few months in Ubuntu 22 so it isn’t like all the issues are gone. I do agree that there are less issues now than there was before, but I will still say that btrfs still breaks or behaves unexpectedly much more often than ext4 or xfs.
I can reliably, across vendors and drives, break RAID10 on BTRFS where MD+LVM are totally fine. Simply pull power. Discovered this when building out my latest workstation.
I haven't tried other configurations; after finding this pattern I decided to leave BTRFS for single-disk configurations where I want CoW
I think you're describing UFS soft-updates? I think that's more or less for meta data updates, not data data. It's been a while since I reviewed it, but it gets you nice things like snapshots and background fsck so after an unclean restart your system can get back to work immediately and clean up behind the scenes. There is some sort of journalling that's fairly new, but my experience from 10 years ago was soft-updates and background fsck just worked; and if you wanted better, ZFS was probably what you want, if you can afford copy on write.
ZFS is used heavily on Linux and runs well, though there are some limitations which are being addressed over time in the OpenZFS project. It is used across all areas that Linux serves, whether laptop, desktop, home server all the way to enterprise. https://openzfs.org/wiki/Main_Page
Meaning unless you want to put in the time to manually test every kernel update and ensure your kernel version stays in-sync with OpenZFS you can very likely end up with an unbootable system.
Most of the SGI machine I've used of various sizes did not have hardware raid. In my experience, you were more likely to run into hardware raid on a PC than on traditional SGI or Sun servers (I don't have much experience with AIX or HP-UX), unless the unix server was in a SAN environment.
(As for hardware raid - I didn't start to see hardware raid regularly until HP started shipping rackmount servers with Compaq raid hardware, way back. Linux boxes..)
Bad old days.