Bug hunting in Btrfs
tavianator.com
tavianator.com
Handrolled concurrency using atomic integers in C. Without the proof system or state machine model to support it. Seems likely that's not going to be their only race condition.
The CSS that sets the background of the code elements (so they properly cover up the ones under them) is attached to the hljs class, which is added to those elements by highlight.js.
No JS > no classname added > no background style > jank
Also the code used to use locks, and it wasn't any simpler: https://lore.kernel.org/linux-btrfs/20230503152441.1141019-2...
NOT_IN_CACHE -> READING -> IN_CACHE
Readers that encounter the READING state would insert a completion handler into a queue, and readers transitioning out of the READING state would wake up all the completion handlers in the queue.
I've been working on an open source library and simple (manual) proof system that makes it easy to verify that the queue manipulation and the state machine manipulation are atomic with respect to each other:
https://docs.rs/atomic-try-update/0.0.2/atomic_try_update/
The higher level invariants are fairly obvious once you show that the interlocks are correct, and showing the interlocks are correct is just a matter of a quick skim of the function bodies that implement the interlocks for a given data type.
I've been looking for good write ups of these techniques, but haven't found any.
But Linux kernel APIs like test_bit(), set_bit(), clear_bit(), test_and_set_bit() etc. only work on one bit at a time. The advantage is they can avoid a CAS loop on many platforms. The disadvantage is you only get atomic transitions for one bit at a time. So the `READING -> UPTODATE` transition is more like
READING -> (READING | UPTODATE) -> UPTODATE
if (!(bflags & UPTODATE)) // atomic
// race could happen here
bflags |= READING; // atomic
My NAS and several larger datasets are on ZFS based on reputation alone, but honestly I've had more data loss scares with ZFS than BTRFS (drives that have disappeared for no reason then reappeared hours later, soft locked and unable to unmount indefinitely, several unfortunate user-error issues with automounting datasets overlaying necessary top-level directories and preventing successful boots), and I find the BTRFS tooling more intuitive.
For my hobbyist-level homelab type needs, I would say I'm overall pretty happy with BTRFS. The only issue I've never been able to resolve is regarding lockups when issuing quotas -- another reason I stick to ZFS for my spinning rust storage drives.
Oh, and the ZFS ability to mount a zvol as a foreign filesystem (words?) lets me `btrfs send` backups to ZFS, which is nice!
You've had 3 SSD failures in just 7 years? Any more details about those disks?
And to be fair, 2 of 3 were only suspicious for having a hardware problem so I replaced with a larger drive (and was super nice to have such a straightforward was to expand the size as compared to ZFS).
When I wanted to reinstall I naively thought I could format the root super volume and keep /home subvolume -- but this was impossible: I had to format /home as well according to the OpenSUSE Tumbleweed installer.
Major problem for me. I now have separate root (btrfs) and home (ext4) partitions.
Separate BTRFS root and ext4 home partitions is either the default filesystem layout now if you're not doing FDE or the second recommended one.
The non-footgun way is to have several subvolumes (@root, @boot, @home for btrfs, rpool/ROOT/system_instance and rpool/USERDATA for zfs), and nothing in else in the volume itself. Then, you wipe the system subvolume and create a new one, with new system. Or just keep the old system subvolume and create a new one.
* Start with an ext3 filesystem 70% full.
* Convert to btrfs using btrfs-convert.
* Delete the ext3_saved snapshot of the original filesystem as recommended by the convert utility.
* Enable compression (-o compress) and defrag the filesystem as recommended by the man page for how to compress all existing data.
It fails with out of disk space, leaving a filesystem which isn't repairable - deleting files will not free any space.
The fact such a bug seems to have existed for years, with such a basic following of the man pages for a common use case (migration to btrfs to make use of its compression abilities to get more free space), tells me that it isn't yet ready for primetime.
Does a rebalance fix it? I have once (and only once, back when it was new) hit a "out of disk space" situation with btrfs, and IIRC rebalancing was enough to fix it.
> for a common use case
It might have been a common use case back when btrfs was new (though I doubt it, most users of btrfs probably created the filesystem from scratch even back then), but I doubt it's a common use case nowadays.
It's perhaps not as common as it once was, but you'd expect it to be common enough to work, and not some obscure corner case.
Same for the RAID5/6 bugs in BTRFS. What's their solution? A simple warning in the docs:
> RAID5/6 has known problems and should not be used in production. [0]
Also the CLI discourages you from creating these things. Brilliant.
This is why I don't use BTRFS anywhere. An FS shall be bulletproof. Errors must only cause from hardware problems. Not random bugs in a filesystem.
[0]: https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html#multi...
Would I build a file storage system around btrfs? No - without proper redundancy at least. But I'm told at least Synology does.
I'm pretty sure there's plenty of cases where it's perfectly usable - the feature set it has today is plenty useful and the worst case scenario is an host reimage.
I can live with that. applications will generally break production ten billion times before btrfs does.
I know, I'm a sysadmin. I care for hardware, mend it, heal it, and sometimes donate, cann-bird or bury it. I'm used to it.
> worst case scenario is an host reimage...
While hosting PBs of data on it? No, thanks.
> Would I build a file storage system around btrfs? No - without proper redundancy at least.
Everything is easy for small n. When you store 20TB on 4x5TB drives, everything can be done. When you have a >5PB of storage on racks, you need at least a copy of that system running hot-standby. That's not cheap in any sense.
Instead, I'd use ZFS, Lustre, anything, but not BTRFS.
> I can live with that - applications will generally break production ten billion times before btrfs does.
In our case, no. Our systems doesn't stop because a daemon decided to stop because a server among many fried itself.
But again, most systems are not like that. Kubernetes cluster nodes? Reimage at will. Compute nodes for vms backed by SAN? Reimage at will. Btrfs can actually make that reimage faster and it's pretty reliable on a single flash drive so why not?
On the other hand, BTRFS’ single disk performance, esp, for small files is visibly lower than EXT4 and XFS, so why bother?
There are many solutions for EXT4 which allows versioning, and if I can reimage a node (or 200) in 5 minute flat, why should I bother with the overhead of BTRFS?
It’s not that I haven’t tried BTRFS. Its features are nice, but from my perspective, it’s not ready for prime time, yet. What bothers me is the mental gymnastics pretending that it’s mature at this point.
It’ll be good file system. An excellent one in fact, but it still needs to cook.
But it has been that way for now 14 years. Sure, there are far fewer sharp edges now than there were back then. For a host you can just reimage it's fine, for a well-tested fairly restricted system it's fine. I stay far away from it for personal computers and my home-built NAS, because just about any other fs seems to be more stable.
I have bitten by ReiserFS' superblock corruption once, and that time I had plenty of time to rebuild my system leisurely. My current life doesn't allow for that. I need to be able to depend on my systems.
Again, I believe BTRFS will be an excellent filesystem in the long run. It's not ready yet for "format, mount and forget" from my perspective. Only I'm against is, "it runs on my machine, so yours' is a skill issue" take, which is harmful on many levels.
Personally, btrfs just works and the features are worth it.
Btrfs raid always gets brought up in these discussions, but you can just not use it. The reality is that it didn't have a commercial backer until now with Western Digital.
If I'm just not gonna use BTRFS' RAID, I can just use mdadm + any file system I want. In this case, any file system becomes "anything but btrfs" from my point of view.
I've burnt by ReiserFS once. I'm not taking the same gamble with another FS, thanks.
The same probably applies to may consumer devices.
This is drastic, and I'd rather perform such an operation on an image copy.
This is one case where ZFS is absolutely superior; if a drive goes offline, and is returned to a set at a later date, the resilver only touches the changed/needed blocks. Btrfs forces the entire filesystem to be rewritten in a rebalance, which is much more drastic.
I am very willing to allow ZFS mirrors to be degraded; I would never, ever let this happen to btrfs if at all avoidable.
I would agree that if this is presented in the documentation as something it supports, then it should work as expected. If it doesn’t work, then a pull request to remove it from the docs might be the best course of action.
All in all not too terrible and I just accepted the bogus data as good by resetting (?) the checksums for those blocks. But acting as if it’s just a terminal command and Bob’s your uncle.. that’s just wrong.
It seems like a reasonable thing to want to do. Would you never update an installed application or the kernel to get new features or fixes? I don't really think there's much fundamental difference. If what you mean is "it seems likely to fail catastrophically", well that seems like an indication that either the converter or the target filesystem isn't in a good state.
This was actually a routine thing to do, under iOS and macOS, with the transition from HFS to APFS.
Honestly, if that was an option without the certainty of getting pwned due to some RCE down the road, then yes, there are cases where I absolutely wouldn't want to update some software and just have it chugging away for years in its present functional state.
Sure there are, sometimes! But at the same time, in other situations stability and predictability would take precedence over anything else, e.g. any new features that might get released probably wouldn't matter too much for a given use case.
For example, I could take a new install of MariaDB that has come out recently and use it for the same project for 5 years with no issues, because the current feature set would be enough for the majority of use cases. Of course, that's not entirely realistic, because of the aforementioned security situation.
Applies the same to OS and/or kernel versions, like how you could take a particular version of RHEL or Ubuntu LTS and use it for the lifespan of some project, although in that case you do at least get security updates and such.
this seems sensible
You are as empowered to fix this as anybody else, if it presents a real problem for you.
If nobody cares that much, maybe deprecate the tool, then?
Also, just because it's Open Source (TM) doesn't mean developers will accept any patch regardless of its quality. Like everything, FOSS is 85% people, 15% code.
> You are as empowered to fix this as anybody else, if it presents a real problem for you.
I have reported many bugs in Open Source software. If I had the time to study code and author a fix, I'd submit the patch itself, which I was able to do, a couple of times.
If you think that's the right thing to do, you're as free as anybody else to send documentation patches. I doubt anybody would argue with you here, but who knows :)
> Also, just because it's Open Source (TM) doesn't mean developers will accept any patch regardless of its quality.
Of course not. If you want to make a difference, you have to put in the work. It's worth it IMHO.
That won't do anything. Instead I can start a small commotion by sending a small request (to the mailing lists) to deprecate the tool, which I don't want to do. Because I'm busy. :)
Also, I don't like commotions, and prefer civilized discussions.
> Of course not. If you want to make a difference, you have to put in the work. It's worth it IMHO.
Of course. This is what I do. For example, I have a one liner in Debian Installer. I had a big patch in GDM, but after coordinating with the developers, they decided to not merge the fix + new feature, for example.
Surely you were given some justification as to why they didn't want to merge it? I realize sometimes these things are intractable, but in my experience that's rare... usually things can iterate towards a mutually agreeable solution.
GTK has (or had) a sliding infoline widget, which is used to show notifications. GDM used it for password related prompts. It actually relayed PAM messages to that widget.
We were doing mass installations backed by an LDAP server which had password policies, including expiration enabled.
That widget had a bug, prevented it displaying a new message when it was in the middle of an animation, which effectively ate messages related to LDAP (Your password will expire in X days, etc.).
Also we needed a keyboard selector in that window, which was absent.
I gave heads up to the GDM team, they sent a “go ahead” as a reply. I have written an elaborate patch, which they rejected and wanted a simpler one. I iterated the way they wanted, they said it passes the muster and will be merged.
Further couple of mails never answered. I’m basically ghosted.
But the merge never came, and I moved on.
I’m getting sick and tired of this argument. It’s very user unfriendly
Do you work in software? If yes, do you own a component? Have you tried using this argument with your colleagues that have no idea about your component?
Of course they theoretically are empowered to fix the bug, but that doesn’t make it easier. They may have no idea about how filesystem internal. Or they may be just users and have no programming background at all
Open source doesn't work for you. You can't demand that it do things you want: you have to push and justify them yourself, and hopefully you find other people along the way who want the same things and can help you.
I taught myself to program as a teenager trying to fix bugs in Linux. If I could do it, anybody can. That's what makes it empowering.
But being open source doesn't mean people shouldn't or can't just say "this is bad" or "don't use x" or even "this X feature doesn't work when using Y" (now If what they are saying isn't true, sure we should call them out)
It's normal and in fact I think it should be encouraged, as not everyone can be aware of potential problems they could have with a piece of OSS that is already known in some obscure dev mailing list. The more people are informed and aware of potential issues, the less they will be surprised and complain about it when they start using it.
(Fwiw I love btrfs, and I think it's very reliable)
Please don’t put words in other peoples mouth.
SSD used: Samsung SSD 970 PRO 1T, same installation since 2020-05-02.
* FWIW, I used native BTRFS RAID5 for years and never had an issue but that's just anecdata
The actual patch is buried at the end the article: https://lore.kernel.org/linux-btrfs/1ca6e688950ee82b1526bb30...
Also, if you don't mind sharing your kconfig that would be interesting to see too.
I did submit a patch to add a WARN_ON() for this case: https://lore.kernel.org/linux-btrfs/d4a055317bdb8ecbd7e6d9bd...
But to catch the general class of bugs you'd need something like KCSAN. I did try that but the kernel is not KCSAN-clean so it was hard to know if any reports were relevant.
Something as general as KCSAN isn't necessary: it's a classic ABA problem, double checking the ->bflags value on transitions is sufficient to catch it. Like a lockdep-style thing where *_bit() are optionally replaced by helpers that check the current value and WARN if the transition is unexpected.
Using the event numbering from your patch description, such a thing would have flagged seeing UPTODATE at (2). But the space of invalid transitions is is much larger than the space of valid ones, which is why I think it might help catch other future bugs sooner.
Dunno if it's actually worth it, but I definitely recall bugs of this flavor in the past. It'll take some work to unearth them all, alas...
Having worked on huge deployments of Linux servers before, I can tell you that modelling race condition bugs as having a uniform random chance of happening per unit time is shockingly predictive. But it's not proof, obviously.
I don't generally disagree with that methodology, but 100 hours is just not a lot of time.
If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed, and yet it will still affect nearly 100% of all of your users after a bit more than a month!
For file systems, the aim should be (much more than) five nines, not nine fives.
Yes. Which means 9/10 users who used their machines 100 or fewer hours on the new kernel will never hit the hypothetical bug. Thank you for proving my point!
I'm not a filesystem developer, I'm a user: as a user, I don't care about the long tail, I only care about the average case as it relates to my deployment size. As you correctly point out, my deployment is of negligible size, and the long tail is far far beyond my reach.
Aside: your hypothetical event has a 0.1% chance of happening each hour. That means it has a 99.9% chance of not happening each hour. The odds it doesn't happen after 100 hours is 0.999^100, or 90.5%. I think you know that, I just don't want a casual reader to infer it's 90% because 1-(100/1000) is 0.9.
No, that's not how probabilities work at all for a bug that happens with uniform probability (i.e. not bugs that deterministically happen after n hours since boot). If you have millions of users, some of them will hit it within hours or even minutes after boot!
> As you correctly point out, my deployment is of negligible size, and the long tail is far far beyond my reach.
So you don't expect to accrue on the order of 1000 machine-hours in your deployment? That's only a month for a single machine, or half a week for 10. That would be way too much for me even for my home server RPi, let alone anything that holds customer data.
> I'm not a filesystem developer, I'm a user: I don't care about the long tail, I only care about the average case as it relates to my deployment size.
Yes, but unfortunately you seem to either have the math completely wrong or I'm not understanding your deployment properly.
The 1000 number came from you. I have no idea where you got it from. I suspect the "real number" is several orders of magnitude higher, but I have no idea, and it's sort of artificial in the first place.
My overarching point is that mine is such a vanishingly small portion of the universe of machines running btrfs that I am virtually guaranteed that bugs will be found and fixed before they affect me, exactly as happened here. Unless you run a rather large business, that's probably true for you too.
The filesystem with the most users has the least bugs. Nothing with the feature set of btrfs has even 1% the real world deployment footprint it does.
> If you have millions of users, some of them will hit it within hours or even minutes after boot!
This is weirdly sensationalist: I don't get it. Nobody dies when their filesystem gets corrupted. Nobody even loses money, unless they've been negligent. At worst it's a nuisance to restore a backup.
It's an arbitrary example of an error rate you'd have a 90% chance of missing in your sample size of 100 machine-hours, yet much too high for almost any meaningful application.
I have no idea what the actual error rate of that btrfs bug is; my only point is that your original assertion of "I've experienced 100 error-free hours, so this is a non-issue for me and my users" is a non sequitur.
> This is weirdly sensationalist: I don't get it. Nobody dies when their filesystem gets corrupted. Nobody even loses money, unless they've been negligent.
I don't know what to say to that other than that I wish I had your optimism on reliable system design practices across various industries.
Maybe there's a parallel universe where people treat every file system as having an error rate of something like "data corruption/loss once every four days", but it's not the one I'm familiar with.
For better or worse, the bar for file system reliability is much, much, much, much higher than anything you could reasonably produce empirical data for unless you're operating at Google/AWS etc. scale.
It's a statement of fact: it has been a non-issue for me. If you're like me, it's statistically reasonable to assume it will be a non-issue for you too. Also, no users, just me. "Proabably okay" is more than good enough for me, and I'm sure many people have similar requirements (clearly not you).
I have no optimism, just no empathy for the negligent: I learned my lesson with backups a long time ago. Some people blame the filesystem instead of their backup practices when their data is corrupted, but I think that's naive. The filesystem did you a favor, fix your shit. Next time it will be your NAS power supply frying your storage.
It's also a double edged sword: the more reliable a filesystem is, the longer users can get away without backups before being bitten, and the greater their ultimate loss will be.
Yes...
> If you're like me, it's statistically reasonable to assume it will be a non-issue for you too.
No! This simply does not follow from the first statement, statistically or otherwise.
You and I might or might not be fine; you having been fine for 100 hours on the same configuration just offers next-to-zero predictive power for that.
> You and I might or might not be fine; you having been fine for 100 hours on the same configuration just offers next-to-zero predictive power for that.
You're missing the forest for the trees here.
It is predictive ON AVERAGE. I don't care about the worst case like you do: I only care about the expected case. If I died when my filesystem got corrupted... I would hope it's obvious I wouldn't approach it this way.
Adding to this: my laptop has this btrfs bug right now. I'm not going to do anything about it, because it's not worth 20 minutes of my time to rebuild my kernel for a bug that is unlikely to bite before I get the fix in 6.9-rc1, and would only cost me 30 minutes of time in the worst case if it did.
I'll update if it bites me. I've bet on much worse poker hands :)
Is that reliable enough for you? Great! Is that “very rare”? Absolutely not for almost any type of user/scenario I can imagine.
If you’re making any statistical arguments beyond that data, or are implying more data than that, please provide either, otherwise this will lead nowhere.
The expected case after surviving a hundred hours is that you're likely to survive another hundred.
Which is a completely useless promise.
That piece of data doesn't let you predict anything at reasonable time scales for an OS install.
You can't squeeze more implications out of such a small sample.
> The expected case after surviving a hundred hours is that you're likely to survive another hundred.
That's exactly right. I don't expect to accrue another hundred hours before the new release, so I'll likely be fine.
> Which is a completely useless promise.
Statistics is never a promise: that's a really naive concept.
> at reasonable time scales for an OS
The timescale of the OS install is irrelevant: all that matters is the time between when the bug is introduced and when it is fixed. In this case, about nine months.
Even so, "likely" here is something like "better than 50:50". Your claim was "very very rare" and that's not supported by the evidence.
> Statistics is never a promise: that's a really naive concept.
It's a promise of odds with error bars, don't be so nitpicky.
You're free to disagree, obviously, but I think it's accurate to describe a race condition that doesn't happen in 100 hours on a multiple machines with clock rates north of 3GHz as "very very rare". That particular code containing the bug has probably executed tens of millions of times on my little pile of machines alone.
> It's a promise of odds with error bars, don't be so nitpicky.
No, it's not. I'm not being nitpicky, the word "promise" is entirely inapplicable to statistics.
So you haven't heard of zfs then?
Even as an anecdote this is completely useless. A couple thousand hours and dozens of mount/unmount cycles would just be a good start.
So....that's really bad.
> If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed, and yet it will still affect nearly 100% of all of your users after a bit more than a month!
I understand the point you're trying to make here, but 1000 is just an incredibly unrealistically small number if we're modelling bugs like that. The real number might be on the order of millions. The effect you're describing in real life might take decades: weeks is unrealistic.
But that makes your evidence of 100 error-free hours even less useful to make any predictions about stability!
You're still conflating probabilities an event occurs across a group with the probability an event happens to one specific individual in that group (in this case, me). I'm talking about the second thing, and it's very much not the same.
If I could rewind the universe and replay it many many times, some portion of those times I will either be very lucky or very unlucky, and get an initial testing result that badly mispredicts my personal future. But we know that most of the time, I won't.
I can actually prove that. Because of the simple assumptions we're making, we can directly compute the probabilities we are initially that wrong:
Odds we test 1000-hour bug for 1000 hours without tripping: 0.999^1000 = 36.7%
Odds we test 50-hour bug for 100 hours without tripping: 0.98^100 = 13.3%
Odds we test 10-hour bug for 100 hours without tripping: 0.9^100 = 0.002%
Of course, in the real world, you might never hit the bug because you have to pat yourself on the head while singing Louie Louie and making three concurrent statx() calls on prime numbered CPUs with buffers off-by-one from 1GB alignment while Mars is in the fifth house to trigger it... it's just a model, after all.
Anecdotal statistics maybe.
Not once would I even insinuate that after my 400 hours of experience, HFS is a file system people should rely on.
100 hours says nothing. When I read your comment I just assumed that you had typoed 100 machine-years (as in running a server farm) as that would have been far more relevant.
[0] Participating in the MARCHintosh GlobalTalk worldwide AppleTalk network full of vintage computers and emulators
I do agree it's "very very rare in practice". You need at least 3 threads racing on the same disk block in a ~15 instruction window, during which time you need the other thread to start and finish reading a block from disk. Which means you need very fast I/O and decryption and enough CPU oversubscription to get preempted in the race window. And it only happens when you actually hit disk which means once your machine is in a steady state, most metadata blocks you need will already be cached and you'll never see the bug.
That said, this is not some crazy 1-instruction race window with preemption disabled. With enough threads calling statx() in parallel and regularly dropping caches, I can reproduce it consistently within a few minutes. Way less than 100 hours :)
It does though: it shows it isn't a dire usability problem. A machine with this bug is still able to do meaningful work, it just might eat it's own tail occasionally.
If I ran N servers with this bug and had an automated thing to whack them when they tripped over it, how many would be out of commission at any given time? All signs point to it being a pretty small number. As long as data loss isn't a concern, a sysadmin would probably choose to wait a couple weeks for the fix through the proper channels, rather than going to a lot of trouble to do it immediately themselves (which also carries risk).
This is also why I was curious about kconfig: my test boxes have all the hardening options turned on right now, I could see how that might make it less likely to trigger.
I completely agree the "uniform random bug" model is only so useful, since an artificial reproducer obviously blows it out of the water. But as I said elsewhere, I've seen it be shockingly predictive when applied to large numbers of machines with bugs like this.
Do snapshots degrade the performance of btrfs?
Unfortunately, deleting snapshots with quotas is an expensive operation that needs to rescan some structures to keep the quota information consistent and that is what you're seeing.
This is 2 bugs really. 1, the file system got corrupted. 2, tooling didn't exist to automatically scan through the disk data structures and recover as much of your drive as possible from whatever fragments of metadata and data were left.
For 2, it should happen by default. Most users don't want a 'disk is corrupt, refusing to mount' error. Most users want any errors to auto-correct if possible and get on with their day. Keep a recovery logfile with all the info needed to reverse any repairs for that small percentage of users who want to use a hex editor to dive into data corruption by hand.
* scan through the whole disk and, for every sector, decide if it is "definitely free space (part of the free space table, not referenced by any metadata)", "definitely metadata/file data", "unknown/unsure (ie. perhaps referenced by some dangling metadata/an old version of some tree nodes)".
* I would then make a new file containing a complete image of the whole filesystem pre-repair, but leaving out the 'definitely free space' parts.
* such a file takes nearly zero space, considering btrfs's copy-on-write and sparse-file abilities.
* I would then repair the filesystem to make everything consistent. The pre-repair file would still be available for any tooling wanting to see what the filesystem looked like before it was repaired. You could even loopmount it or try other repair options on it.
* I would probably encourage distros to auto-delete this recovery file if disk space is low/after some time, since otherwise the recovery image will end up pinning user data to using up disk space for years and users will be unhappy.
The above fails in only one case: Free space on the drive is very low. In that case, I would probably just do the repairs in-RAM and mount the filesystem readonly, and have a link to a wiki page on possible manual repair routes.
No. Most of the block will be marked as unsure in first step -- because most of them had been used before thanks to CoW
It's understandable that corruption can happen due to bugs or hardware failure or user insanity, but my experience was that the recovery tools are useless, and that's a big problem.
I have also done this, and was also happy that the only corruption was to a handful of unimportant log files. Part of a robust filesystem is that when the user does something stupid, the blast radius is small.
Other less-smart filesystems could easily have said "root of btree version mismatch, deleting bad btree node, deleting a bunch of now unused btree nodes, your filesystem is now empty, have a nice day".
If possible include the last kernel log entries before it corrupted. Include kernel version, drive model and drive firmware version.
Worst corruption was when one DIMM started corrupting data. As a result computer kept crashing and eventually refused to mount because of btrfs checksum mismatches.
Fix was to buy new HW. Then run btrfs filesystem repairs, which failed at some point but at least got the filesystem running as long as I did not touch the most corrupted locations, luckily it was RAID1 so most checksums had a correct value on another disk. Unfortunately the checksum tree had on two locations corruption on both copies. I had to open the raw disks with hex editor and change the offending byte to correct value, after which the filesystem has been running again smoothly for 5 years.
And to find the location to modify on the disks I built a custom kernel that printed the expected value and absolute disk position when it detected the specific corruption. Plus had to ask a friend to double check my changes since I did not have any backups.
So did you bite the bullet and get ECC, or are you just waiting for the next corruption caused by memory errors? :)
Ext4 is the most reliable file system I have ever used. Just works and has never failed on me, not even once. No idea why btrfs can't match its quality despite over a decade of development.
Edit: Also there was a time when it was normal for databases to be running on Solaris, in which case ZFS is expected; my recollection is that Sun even marketed how great of a combination that was.
Thanks for reporting!
Sunhad some recommendations for tuning PostgreSQL performance on ZFS, but the recommendations seemed weird or even wrong for OLTP on top of CoW filesystem (I assume that these recommendations were meant for the above mentioned DWH/archival scenarios).
https://www.oracle.com/technetwork/server-storage/solaris10/...
i think you can tweak it to make high random write load less painful but it generally will struggle with that.
Those replicas are not used for any workload, besides nightly consistency checks for MySQLs via pt-table-checksum to ensure we don't have data drift.
Snapshots are crash consistent. Once in a while they give us ability to very quickly inspect how data looked like few minutes or hours ago. This can be life-saver in case of fat-fingering a production data and saved us from lenghty grepping of backups when we needed to recover few records from a specific table.
Yes, I know soft deletes, audit logs - all of those could help and we do have them, but sometimes that's not enough or not feasible.
Due to it's COW nature BTRFS is far from perfect for data that changes all the time [ databases busy with writes, images of VMs with plenty of disk write activity ]. There's plenty of write amplification, but that can be solved with NVMe drives thrown on the problem.
It ought to be easy to design a filesystem which can have pretty much zero overhead for those two operations. I'm kinda disappointed that every filesystem doesn't perform identically for the database workload.
I totally understand that different file systems would do different tradeoffs affecting directory listing, tiny file creation/deletion, traversing deep directory trees, etc. But random reads of a huge file ought to perform near identically to the underlying storage medium.
A stable clean-room ZFS with on-disk compatibility would be a huge task. How long did stable NTFS write capability take? And NTFS is a much simpler filesystem. It would also be a huge waste of time given that btrfs and bcachefs exist, and that ZFS is fine to use license-wise – it's just distribution that's a tad awkward (but not overly so).
https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
Maybe it was Mason's pet project within the company, but there is no ambiguity that Oracle owns it. It is an Oracle project.
But does it even do that? You might be surprised when/if you read a little more widely. Position of OpenZFS project[0] is which I find persuasive (my emphasis added):
In the case of the Linux Kernel, this prevents us from distributing OpenZFS as part of the Linux Kernel binary. *However, there is nothing in either license that prevents distributing it in the form of a binary module* or in the form of source code.
You might see also:
[1]: https://www.networkworld.com/article/836039/smb-encouraging-...
[2]: https://law.resource.org/pub/us/case/reporter/F2/977/977.F2d...
Might we agree then that the only thing that really matters is the law? And we should ignore other opinions coughlike from the FSF/the SFCcough which don't make reference to the law? Or which ignore long held copyright law principles, like fair use?
Please take a look at the case law. The so far theoretical claim of OpenZFS/Linux incompatibility is especially weak re: a binary kernel module.
Veer in what direction? The current state of affairs re: Canonical and Oracle is a ZFS binary kernel module shipped with the Linux kernel. Canonical has literally done the thing that which you are speculating is impermissible. And, for 8 years, Oracle has done nothing. Oracle's attorneys have even publicly disagreed with the SFC that a ZFS binary kernel module violates the GPL or the CDDL.[0]
Given this state of affairs, the level of legal certainty re: this question is far greater than the legal certainty we have re: pretty much any other open IP question in tech.
What matters is practice, is that you stop your/the SFC's/the FSF's torrent of FUD.
> Even a complete rewrite of ZFS would have some risk associated with it given Oracle's history.
I'd ask "How?", but it'd be another torrent of "What if..."s.
In case my lawnmower reference confused you enough that you were unable to make an appropriate response to the point I actually made, I'll try to state it a bit more clearly:
It does not matter how confident you are that Oracle would eventually lose a lawsuit over using or distributing ZFS with the Linux kernel. If Oracle decides to attempt to exert control over ZFS and interfere with the use or distribution of ZFS with Linux, they have ample resources to make a lot of very expensive trouble for various users and organizations. Oracle's history—most importantly, their history vs Google re Java in Android—means it would not be much of a stretch for them to decide to start behaving like The SCO Group. I do not think this risk is large. But I do think it is a real risk that a cautious Linux distro can reasonably be worried about.
If you truly believe that Oracle's lack of action thus far against ZFS on Linux and their public statements of their beliefs about the effects of the CDDL and GPL would prevent them from starting shit, then you are simply wrong about how our legal system works, and there are plenty of examples. The things you point to to bolster your arguments about what the law says are things that would make it hard for Oracle to win a lawsuit on its merits, but the eventual judgement is hardly the only thing that matters when assessing a legal risk—especially if your pockets are not as deep as Oracle's.
The 2nd or 3rd most commercially important Linux distro has been using ZFS since 2016.
> If you truly believe that Oracle's lack of action thus far against ZFS on Linux and their public statements of their beliefs about the effects of the CDDL and GPL would prevent them from starting shit
I understand the argument "Oracle might do something", plus spooky magic fingers and creepy noises, all too well. Except it's not actually an argument. It might be best described as a boogeyman, sent to frighten little children into not running ZFS.
My point was: I think it's time for you to get over sleeping with the light on.
In 2000, we called this FUD when Microsoft did this. In 2024, we should know better, even when you're fronting for the FSF or the SFC.
There's also the fact that not everything is OOB compatible with ZFS. For example, newer versions of systemd have been able to use the TPM to unlock drives encrypted with LUKS. AFAIK it doesn't work with ZFS.
I use ZFS on my daily driver Linux box and mostly love it, but as long as these things happen, I can see why people may want to try to find an in-kernel solution. I personally use Arch so expect for bleeding edge updates to not work perfectly right away. But I recall seeing folks complaining about issues on Fedora, too, which I expect to be a bit more conservative than Arch.
But presumably you meant native ZFS encryption, which unfortunately is considered to be somewhat experimental and neglected, as I understand. Which surprised me, since I thought data at rest encryption would be pretty important for an "enterprise" filesystem
Still, apparently lots of people successfully run ZFS on LUKS. It does mean you don't get zero-trust zfs send backups, but apparently that's where a lot of the bugs have been anyway
> But presumably you meant native ZFS encryption, which unfortunately is considered to be somewhat experimental and neglected, as I understand. Which surprised me, since I thought data at rest encryption would be pretty important for an "enterprise" filesystem.
Yeah, I've happened about someone saying something similar, but I've never seen anything about that from "official" sources. Wouldn't mind a link or something if you have one on hand. But there is the fact that this encryption scheme seems limited when compared to LUKS: there's no support for multiple passphrases or using anything more convenient, like say, a U2F token.
> Still, apparently lots of people successfully run ZFS on LUKS. It does mean you don't get zero-trust zfs send backups, but apparently that's where a lot of the bugs have been anyway
I'd say I'm one of those people, never had any issue with this in ~ten years of use. And, indeed, the main reason for using this on my laptop is being able to send the snapshots around without having to deal with another tool for managing encryption. Also, on my servers on which I run RAIDZ, having to configure LUKS on top of each drive is a PITA.
I remember when Btrfs was initially released (I was doing Solaris sysadmining, and it was billed as "ZFS for Linux"), and yet here we are all these years later and it still seems to be 'meh'.
† E.g., RAID5+ that's less likely to eat data:
* https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...
* https://bcachefs.org/ErasureCoding/ / https://github.com/koverstreet/bcachefs/issues/657
For your specific example, bcachefs's erasure coding is very experimental and currently pretty much unusable, while btrfs is actively working towards fixing the raid56 write hole with the recent addition of the raid-stripe-tree. By the time bcachefs has a trustworthy parity profile, btrfs's may be just as good.
My specific example says that the bcachefs are "actively working towards fixing the raid56 write hole" as well—or rather, their way of doing things doesn't have one in the first place.
Yep, that's my point. Neither btrfs nor bcachefs have a write-hole-less parity raid profile implementation yet, and both are working towards one. We don't know if one will be finished and battle tested significantly before the other, or if one will prove to be more performant or reliable. Just have to wait and see.
* https://packages.debian.org/search?keywords=linux-image-6.7
* Search "6.7.1-1~exp1": https://metadata.ftp-master.debian.org/changelogs//main/l/li...
Until now (bcachefs has been merged) it has only been used by people running custom kernels. As more people try it, it will hit more and more drives with buggy firmware and whatnot.
(Somewhat related, but there has been a WIP 6.1 kernel for WSL2 "in preview" for a while now... I wonder why it hasn't become the default considering both it and 5.12 are LTS... For filesystems like btrfs I often want a newer kernel to pick up every bugfix.)
Can you give or link to a brief description of how blksnap works?
Blksnap is better because it does not require setup in advance like LVM, and it does not require interrupting any live users like fsfreeze. It "should" just work with the live block device within the WSL distro.
LWN covered the v1 posting, which is a good start: https://lwn.net/Articles/914031/
This is all somewhat future-looking, and if you only want file-level snapshotting instead of block-level, it's probably easier to try and get btrfs/bcachefs/nilfs2 instead. My WSL2 on Windows 11 shows btrfs present inside /proc/filesystems.
In the 2024 century the default LVM setup still does not offer to leave some spare space on the volume group. Without that unallocated space it's impossible to create snapshots for LVs.
Oh, and the change-tracking function is non existent for both LVM and dev mapper!
https://patchwork.kernel.org/project/linux-block/cover/20240...
And ZFS is actually good. I'm happy with it. I don't think about it. I've moved on.
Sorry, btrfs, but I don't think it's ever going to work out between us. Maybe in a different life.
ZFS is too important to not be in Linux. Yet it's 2024 and it's still not in there. Because of licensing copyright nonsense. The blame falls solely on the copyright owners.
> And ZFS is actually good. I'm happy with it. I don't think about it. I've moved on.
> Sorry, btrfs, but I don't think it's ever going to work out between us. Maybe in a different life.
ZFS has had a worse bug not even 4 months ago. They are both very good filesystems, bugs happen
It's been really good for me. And btrbk is the best backup solution I've had on Linux, btrfs send/receive is a lot faster than rsync even when sending non-incremental snapshots.
(Of course, backing up to another disk is absolutely still needed, but you probably need it less than you think.)
That's what DKMS is for, which most distros use for ZFS. Install and forget.