Hacker News Reader

Comment by danw1979 | Hacker News Reader

danw1979

As a long time sponsor of Kent's on Patreon - $10 a month since 2018, $790 total - I've found this bcachefs debacle really depressing.

I'm not even a bcachefs user, but I use ZFS extensively and I _really_ wanted Linux to get a native, modern COW filesystem that was unencumbered by the crappy corporate baggage that ZFS has.

In the comments on HN around any bcachefs news (including this one) there are always a couple throwaway accounts bleating the same arguments - sounding like the victim - that Kent frequently uses.

To Kent, if you're reading this:

From a long time (and now former) sponsor: if these posts are actually from you, please stop.

Also, it's time for introspection and to think how you could have handled this situation better, to avoid having disappointed those who have sponsored you financially for years. Yes, there are some difficult and flawed people maintaining the kernel, not least of which Linus himself, but you knew that when you started.

I hope bcachefs will have a bright future, but the ball is very clearly in your court. This is your problem to fix.

(I'm Daniel Wilson, subscription started 9th August 2018, last payment 1st Feb 2025)

View on HN

I am also a Patreon supporter, and I intentionally didn't switch to bcachefs until it was merged into the kernel. After all, Linus would never break userspace right?

I am also frustrated by this whole debacle, I'm not going to stop funding him though Bcachefs is a solid alternative to btrfs. It's not at all clear to me what really happened to make all the drama. A PR was made that contained something that was more feature-like than bugfix-like, and that resulted in a whole module being ejected from the kernel?

I really wish, though that DKMS was not such a terrible a solution. It _will_ break my boot, because it always breaks my boot. The Linux kernel really needs a stable module API so that out-of-tree modules like bcachefs are not impossible to reliably boot with.

I also waited until bcachefs was in the mainline. And I have been loving it. In fact, I even have multiple systems using it as root. Rock solid.

DMKS is not going to work for me though. Some of the distros I use do not even support it. Chimera Linux uses ckms.

As for how we got here, It was not just one event. Kent repeatedly ignored the merge window, submitting changes too late. This irked Linux. When Linus complained, Kent attacked him. And Kent constantly ran down the LKML and kernel devs by name (especially the btrfs guys). This burned a lot of bridges. When Linus pushed Kent out, many people rushed to make it permanent instead of rushing to his defense. Kent lost my support in the final weeks by constantly shouting that he was a champion for his users while doing the exact opposite of what I wanted him to do. I want bcachefs in the kernel. Kent worked very hard to get it pushed out.

It really is a great file system though.

> Kent lost my support in the final weeks by constantly shouting that he was a champion for his users while doing the exact opposite of what I wanted him to do.

Honestly I don't think it's just you, I'm the same boat. I find it hard to believe that the entirely predictable outcome of the current situation is what more than 1% of users wanted.

yeah I'm probably going to have to start building my own kernel

I need to write up a proper patreon post on all this stuff, because there's a lot of misinformation going around.

No, I was not "ignoring the merge window". Linus was trying to make and dictate calls on what is and is not a critical bugfix, and with a filesystem eating bug we needed to respond to, that was an unacceptable situation.

Finding defects is a good thing, and fixing defects is a good thing. Adding new features can be a good thing as long as it doesn't introduce or uncover too many new defects in previously stable code. But what is lacking in your development process that you keep finding "critical" defects that could affect a large number of users during the merge window?

It seems like bcachefs would benefit from parallel development spirals, where unproven code is guarded by experimental flags and recommended only for users who are prepared to monitor and apply patches outside of the main kernel release cycle, while the default configuration of the mainline version goes through a more thorough test and validation process and more rarely sees "surprise" breakage.

It certainly appears that Linus can't tell the difference between your new feature introductions and your maintenance fixes, and that should trigger some self-reflection. He clearly couldn't let all of the thousands of different kernel components operate in the style you seem to prefer for bacachefs.

> It seems like bcachefs would benefit from parallel development spirals, where unproven code is guarded by experimental flags and recommended only for users who are prepared to monitor and apply patches outside of the main kernel release cycle, while the default configuration of the mainline version goes through a more thorough test and validation process and more rarely sees "surprise" breakage.

Maybe if you're a distance observer who isn't otherwise participating in the project

What's the saying about too many cooks in the kitchen?

These concerns aren't coming from the people who are actually using it. They're coming from people who are accustomed to the old ways of doing things and have no idea what working closely with modern test infrastructure is like.

There wasn't anything unusual about the out-of-merge-window patches I was sending Linus except for volume, which is exactly what you'd expect in a rapidly stabilizing filesystem. If anything I've been more conservative about what I send that other subsystems.

> It certainly appears that Linus can't tell the difference between your new feature introductions and your maintenance fixes, and that should trigger some self-reflection. He clearly couldn't let all of the thousands of different kernel components operate in the style you seem to prefer for bacachefs.

If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.

Concerns about general policy need to be handled separately from individual cases. You needed to play along, advocate for your position, explain your position (e.g. demonstrate how your processes eliminate certain classes of errors; or find some objective way to measure QA problems and present the stats), and push for a holistic process change that supports your use-case. That process change would need input from other people, and might end up very different to your original proposal, but it could have happened. Instead, you've burned (and continue to burn) bridges.

Linus's job is not to make the very next version of the kernel as good as it can be. It's to keep the whole system of Linux kernel maintenance going. (Maintaining the quality of the next version is almost a side-effect.) Asking him to make frequent exceptions to process is the equivalent of going "this filesystem is making poor decisions: let's hex-edit /dev/sda to allocate the blocks better". Your priority is making the next version of bcachefs as good as it can be, and you're confident that merging your patchsets won't break the kernel, but that's entirely irrelevant.

> If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.

You have missed the point by a mile.

> Concerns about general policy need to be handled separately from individual cases.

Citation needed.

> Linus's job is not to make the very next version of the kernel as good as it can be. It's to keep the whole system of Linux kernel maintenance going. (Maintaining the quality of the next version is almost a side-effect.) Asking him to make frequent exceptions to process is the equivalent of going "this filesystem is making poor decisions

You're arguing from a false premise here. No exceptions were needed or required, bcachefs was being singled out because he and the other maintainers involved had no real interest in the end goal of getting a stable, reliable, trustworthy modern filesystem.

The discussions, public and private - just like you're doing here - always managed to veer away from engineering concerns; people were more concerned with politics, "playing the game", and - I'm not joking here - undermining me as maintainer; demanding for someone else to take over.

Basic stuff like QA procedure and how we prioritize patches never entered into it, even as I repeatedly laid that stuff out.

> > If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.

> You have missed the point by a mile.

No, that is very much the point here. bcachefs has always had one of the better track records at avoiding regressions and quickly handling them when they do get through, and was being singled out as if something was going horribly awry anyways. That needs an explanation, but one was never given.

Look, from the way you've been arguing things - have you been getting your background from youtube commentators? You have a pretty one sided take, and you're pushing that point of view really hard when talking to the person who's actually been in the middle of it for the past two years.

Maybe you should reevaluate that.

Bureaucracies run on process. You call that "politics", but (when arguing with programmers) I call it code. The Linux kernel project is a bureaucracy.

> citation needed

I'm sure you're familiar with "separation of concerns" in programming: it's the same principle. My experience of bureaucracies is experience, but I'm sure most good books on the topic will have a paragraph or chapter on this.

> bcachefs was being singled out because he and the other maintainers involved had no real interest in the end goal of getting a stable, reliable, trustworthy modern filesystem.

I imagine they would dispute that.

> No exceptions were needed or required,

I know that Linus Torvalds would dispute that. In this HN thread and elsewhere, you've made good arguments that treating your approach as an exception is not warranted, and that your approach is better, but you surely aren't claiming that your approach is the usual approach for kernel development?

> undermining me as maintainer

Part of the job is dealing with Linus Torvalds. You're not good at that. It would make sense for you to focus on architecture, programming, making bcachefs great, and to let someone else deal with submitting patches to the kernel, or arguing those technical arguments where you're right, but aren't getting listened to.

People are "more concerned with politics" than with engineering concerns because the problem is not with your engineering.

> bcachefs has always had one of the better track records at avoiding regressions and quickly handling them when they do get through

That's not relevant. I know that you don't see that it's not relevant: that's why I'm saying it's not relevant.

> have you been getting your background from youtube commentators?

No, but I'm used to disputes of this nature, and I'm used to dealing with unreasonable people. You believe that others are being unreasonable, but you're not following an effective strategy for dealing with unreasonable people. I am attempting to give advice, because I want bcachefs in the kernel, and I haven't a hope of changing Linus Torvalds' mind, but I have half a hope of changing yours.

Rule one of dealing with unreasonable people is to pick your battles. How many times have I said "dispute" or "argument" in this comment? How many times have these been worthy disputes? Even if I've completely mischaracterised the overall situation, surely you can't claim that every argument you've had on the LKML or in LWN comments has been worth bcachefs being removed from the kernel.

Look, it wasn't just this one thing. There had been entirely too many arguments over bugfixes, with the sole justification on Linus's end being "it's experimental garbage that no one should be using".

I can't ship and support a filesystem under those circumstances.

The "support" angle is one you and a lot of other people are missing. Supporting it is critical to stabilizing, and we can't close the loop with users if we can't ship bugfixes.

Given the past history with btrfs, this is of primary concern.

You've been looking for a compromise position, and that's understandable, but sometimes the only reasonable compromise is "less stupidity, or we go our separate ways".

The breaking point was, in the private maintainer thread, a page and a half rant from Linus on how he doesn't trust my judgement, and then immediately after that another rant on how - supposedly - everyone in the kernel community hates me and wants me gone.

Not joking. You can go over everything I've ever said on LKML, including the CoC incident, and nothing rises to that level. I really just want to be done with that sort of toxicity.

I know you and a lot of other people wanted bcachefs to be upstream, and it's not an ideal situation, but there are limits to what I'm willing to put up with.

Now, we just need to be looking forward. The DKMS transition has been going smoothly, more people are getting involved, everyone's fully committed to making this work and I still have my funding.

It's going to be ok, we don't need to have everything riding on making it work with the kernel community.

And now, you guys don't have to worry about me burning out on the kernel community and losing my last vestiges of sanity and going off to live in the mountains and herd goats :)

> Linus was trying to make and dictate calls on what is and is not a critical bugfix, and with a filesystem eating bug we needed to respond to, that was an unacceptable situation.

That's literally his job?

His job is to create drama, and argue over things that don't need to be argued?

Linus has never once found a mistake in the bcachefs pull requests, but he has created massive headaches for getting bugfixes out to users that were waiting on them.

this crowd is wild. the author answer is voted down :)

edit: by the time i commented it was already dark text. guess it recovered.

if you read the LKML archives closely, you'll find this sort of reply from koverstreet (deflecting all responsibility while taking none) as typical as it is deeply misguided.

Just looking for somewhere to stick that pitchfork, eh?

this was my first time speaking publicly about my opinion on this after reading as much of the history as i could with an open mind. i was really looking forward to bcachefs landing in the mainline kernel because btrfs doesn't meet my needs. i was and still am rooting for you, but your social skills really are comically bad and it's impacting the potential of the project. it makes me sad. P.S. your response here doesn't surprise me at all and is completely in character based on everything else i've seen from you.

>It's not at all clear to me what really happened to make all the drama. A PR was made that contained something that was more feature-like than bugfix-like, and that resulted in a whole module being ejected from the kernel?

This isn't just a one time thing, speaking as someone who follows the kernel, apparently this has been going on pretty much since bcachefs first tried to get into Linus's tree. Kent even once told another kernel maintainer to "get your head examined" and was rewarded with a temporary ban.

Edit: To be fair, the kernel is infamous for being guarded by stubborn maintainers but really I guess the lesson to be learned here is if you want your pet project to stick around in the kernel you really can't afford to be stubborn yourself.

> this project sorely needs more than one person involved.

I hold out some hope that somebody else will get involved in bcachefs and that the new person will be able to resubmit bcachefs to the mainline.

My impression is that many people respect the technology and would be happy to have it back--they just cannot work with Kent.

But that is the reasons this will probably not happen. It does not appear that anybody wants to work with Kent.

> I guess the lesson to be learned here is if you want your pet project to stick around in the kernel you really can't afford to be stubborn yourself.

Amen.

And to your point about it being a "pet project" - I'm sure I could go look at the commit history, but is anyone other than Kent actually contributing meaningfully to bcachefs ? If not, this project sorely needs more than one person involved.

> I intentionally didn't switch to bcachefs until it was merged into the kernel. After all, Linus would never break userspace right?

It was explicitly marked experimental.

Experimental, to me, means "it might eat your data, have backups" not "we might decide to remove this module from the kernel for non-technical reasons, good luck users"

Reiserfs sat in the kernel for years after he went to prison and didn't get removed on such short notice even though it was equally if not more unmaintained.

> Experimental, to me, means "it might eat your data, have backups" not "we might decide to remove this module from the kernel for non-technical reasons, good luck users"

I don't think the kernel devs share that definition.

> Reiserfs sat in the kernel for years after he went to prison and didn't get removed on such short notice even though it was equally if not more unmaintained.

Reiserfs wasn't marked experimental.

Although there are "experimental" labels on features in the kernel, there's no coherent definition for what that means. Historically most distros have enabled CONFIG_EXPERIMENTAL features by default, until the flag was deemed meaningless and removed. Linux has never removed a mainline-merged feature this quickly. Features in staging may be removed quickly, but bcachefs was merged into mainline. The bcachefs removal, not to mention the removal timeline, is unprecedented for a mainline-merged feature. There's never been any expectation that experimental means it may be removed on short notice.

Going forward, I'll agree with you: mainline does not care about users of experimental features. But it's disingenuous to suggest that this has always been the expectation.

The biggest mistake is not having a staging subtree for filesystems like we do for most other drivers.

Oh, that's interesting. Is there any reason it couldn't have just gone under drivers/staging?

I think it's mostly a policy thing? I brought it up a few years ago and the fs developers were not very enthused about the idea.

> After all, Linus would never break userspace right?

It was marked experimental. No promises were made.

> It's not at all clear to me what really happened to make all the drama. A PR was made that contained something that was more feature-like than bugfix-like, and that resulted in a whole module being ejected from the kernel?

If you followed the whole saga, Kent had a history of being technically excellent but absolutely impossible to work with.

He did not follow the kernel development process over and over again, instead choosing to post long rants and personal attacks to the mailing lists. Instead of accepting that he has to adapt, he always had (still has) the attitude that he knows best, everybody else are idiots and should just accept his better judgement. Of course the rules should apply… to anyone but him, because his work is special.

Moreover, this was not a one PR thing. Linus actually had a long history of defending Kent, being patient, bending the rules, and generally putting up with him.

This is unambiguously 100% Kent’s fault, which is very sad, because — as mentioned before — Kent’s work is technically excellent. He had a lot of good will that he squandered in spectacular fashion.

> After all, Linus would never break userspace right?

But bcachefs never lived in userspace even before it was merged

In my opinion any family lives in user space, through a implicit contract of filesystems and data stored on disk?

Applications do not talk to the filesystem directly, they talk to the generic I/O syscalls on the kernel which handles the internal filesystem calls.

Those generic syscalls are (supposed to) don't change, the internal filesystem calls can and do change.

This is one reason why ZFS regularly breaks, on top of it can't use GPL exports.

> Applications do not talk to the filesystem directly,

Sometimes, they do. For instance, BTRFS_IOC_CLONE to do a copy-on-write clone of a file's contents (now promoted to other filesystems as FICLONE, but many other ioctl operation codes are still btrfs-specific; and other filesystems have their own filesystem-specific operations).

> A PR was made that contained something that was more feature-like than bugfix-like, and that resulted in a whole module being ejected from the kernel?

From what I can tell, what happened is that was just a trigger for a clash of personalities between Linus and Kent, both of whom have a bit of a temper, and refused to back down, which escalated to this.

What about btrfs?

Seems to tick all of the boxes in regard to what you're looking for, and its mature enough that major linux distros are shipping with it as the default filesystem.

Because every time btrfs is mentioned, 5 more people come out of the woodwork saying that it irreparably lost all their data. Sorry but there's just too many stories for it to be mere coincidences.

Your statement is misleading. No one is using btrfs on servers. Debian and Ubuntu use ext4 by default. RHEL removed support for btrfs long ago, and it's not coming back:

> Red Hat will not be moving Btrfs to a fully supported feature. It was fully removed in Red Hat Enterprise Linux 8.

AFAIK, Facebook uses BTRFS on their servers.

They do, but this is misleading due to a number of caveats

First one is that they don't use btrfs own RAID (aka btrfs-raid/volume management). They actually use hardware RAID so they don't experience any of the stability/data integrity issues people experience with btrfs-raid. Ontop of this, facebooks servers run in data centers that have 100% electricity uptime (these places have diesel generators for backup electricity)

Synology likewise offers btrfs on their NAS, but its underneath mdadm (software RAID)

The main benefit that Facebook gets from btrfs is transparent compression and snapshots and thats about it.

In my experience, btrfs just doesn't seem to be very resilient to hardware faults. Everything works great as long as you stay on the golden path, but when you fall off that path, it gets into a confused state and things start going very wrong and there is no way to recover (short of wiping the whole filesystem, because fsck doesn't fix the faults).

So yes, if you are Facebook, and put it on a rock-solid block layer, then it will probably work fine.

But outside of the world of hyperscalers, we don't have rock solid block layers. [1] Consumer drives occasionally do weird things and silently corrupt data. And on top of drives, nobody uses ECC memory and occasionally weird bit flips will corrupt data/metadata before it's even written to the disk.

At this point, I don't even trust btrfs on a single device. But the more disks you add to a btrfs array, the more likely you are to encounter a drive that's a little flaky.

And Btrfs's "best feature" really doesn't help it here, because it encourages users to throw a large number of smaller cheap/old spinning drives at it. Which is just going to increase the chance of btrfs encountering a flaky drive. The people who are willing to spend more money on a matched set of big drives are more likely to choose zfs.

The other paradox is that btrfs ends up in a weird spot where it's good enough to actually detect silent data corruption errors (unlike ext4/xfs and friends where you never find out your data was corrupted), but then it's metadata is complex and large enough that it seems to be extra vulnerable to those issues.

---------------

[1] No, mdadm doesn't count as a rock-solid block layer, it still depends on the drives to report a data error. If there is silent corruption, madam just forwards it. I did look into using a synology style btrfs on mdadm setup, but I searched and found more than a few stories from people who's synology filesystem borked itself.

In fact, you might actually be worse off with btrfs+mdadm, because now data integrity is done at a completely different layer to data redundancy, and they don't talk to each other.

In a scenario where they don't have to worry about data going poof because it's used to run stateless containers (taking advantage of CoW to reduce startup time etc)

For a long time they were running MySQL on it iirc (outsider, just asked at meetups etc )

Ex-Meta employee here, and yup — this is true.

And they almost always 'forget' to mention "that was in 2010" or "I was using the BTRFS feature marked 'do not use, unstable'".

It's really difficult to get a real feel for BTRFS when people deliberately omit critical information about their experiences. Certainly I haven't had any problems (unless you count the time it detected some bitrot on a hard drive and I had to restore some files from a backup - obviously this was in "single" mode).

My fairly recent experience with some timelines, posted 20d ago: https://news.ycombinator.com/item?id=45210911

Some of the most catastrophic ones were 3 years ago or earlier, but the latest kernel bug (point 5) was with 6.16.3, ~1 month ago. It did recover, but I already mentally prepared to a night of restores from backups...

One of the PostgreSQL devs managed to corrupt Btrfs about 2 years ago when working on async IO. Is that recent enough?

Synology exclusively uses BTRFS afaik, and there aren't widespread stories of data loss with their products.

Took 10 seconds to find:

https://philip.greenspun.com/blog/2024/02/29/why-is-the-btrf...

> We had a few seconds of power loss the other day. Everything in the house, including a Windows machine using NTFS, came back to life without any issues. A Synology DS720+, however, became a useless brick, claiming to have suffered unrecoverable file system damage while the underlying two hard drives and two SSDs are in perfect condition. It’s two mirrored drives using the Btrfs file system

Synology does not use vanilla btrfs, they use a modified btrfs that runs over mdraid mirror, which somehow communicates with btrfs layer to supposedly fix errors, when they occur. It's not clear how far behind that fork is.

Synology are still shipping kernel 5.10 on their latest model. And 4.4 only a few years prior.

I am hoping we will get ZFS from Ubnt NAS via update.

Thats because they use mdadm for the RAID, the btrfs sits underneath a virtual mdadm volume ;)

And also, I've read plenty enough about how hard it has been to maintain btrfs over the years. It's never really felt like the future.

Plus I needed zvols for various applications. I've used ZFS on BSD for even longer so when OpenZFS reached a decent level of maturity the choice between that and btrfs was obvious for me.

The argument of zvols doesn't really fit in here Unless bcachefs supports them?

Not really data loss per se, but let me add my own story to the pile: just last week, I had a btrfs filesystem error out and go permanently read-only simply due to the disk becoming full. Hours of searching and no solution to be found, had to be reformatted.

I don't understand how btrfs is considered by some people to be stable enough for production use.

I know somebody is going to say otherwise, but BTRFS seems genuinely rock solid in single-disk setups. OpenSUSE defaults to it so I've been using it for years. No problems, it's not even something I worry about.

Allowing btrfs to run out of space is well known to do irreparable damage.

Keeping it healthy means paying close attention to "btrfs fi df" and/or "fi usage" for best results.

ZFS also does not react well to running out of space.

I've been running Btrfs on Fedora for a decade now (and it's been the default since 2020). I have basically never done any of those things and it's been fine. I've had to do more babysitting with my ZFS systems than I did my Btrfs ones.

I think a lot of people interested in bcachefs were people who had lost faith in btrfs.

Isn't OpenSUSE still shipping with btrfs on by default?

The rolling "tumbleweed" variant does, afaict there aren't many issues related to btrfs[0]. Most problems I see seem to be Nvidia drivers or something choking during the update process (bad mirrors, odd package not updating, etc.).

[0]I'm currently evaluating OpenSuse as a possible W11 replacement, but not using it for anything serious atm.

I used to run tumbleweed with btrfs, then it lost its root filesystem twice, now I distrust btrfs.

Why don't you know anything about the history of btrfs? You would know if you knew anything, so why don't you know?

No encryption, and I really don't want to do LUKS gymnastics.

> I _really_ wanted Linux to get a native, modern COW filesystem

Doesn't btrfs fit that description? I know there are some problems with it, but it is definitely a native COW filesystem, abd AFAIK it is "modern".

Btrfs has a "happy path" so long as you don't use any features outside of the happy path, your data will generally be fine. Outside of that, your data is less reliably fine.

Btrfs also has issues with large numbers of snapshots, you have to cull them occasionally or things begin to slow down, bcachefs does not.

> In the comments on HN around any bcachefs news (including this one) there are always a couple throwaway accounts bleating the same arguments - sounding like the victim - that Kent frequently uses.

And every time something like this comes up, I end up with every sort of accusation pointed at me, and no one seems to be willing to look at the wider picture - why is the kernel community still unable to figure out a plan to get a trustworthy modern filesystem?

> This is your problem to fix.

No, I've said from the start that this needs to be a community effort. A filesystem is too big for one person.

Be realistic :) If the community wants this to happen, the community will have to step up.

Look, if you're here saying "the throwaway comments aren't me" then I beleive you. It'd be nice if you said that clearly though.

Please please don't forget I want you to succeed - that's why I bunged nearly $800 your way in this endeavour - but I'm not the only person who thinks you come across as completely immune to critisicm, even when it's constructive and from your supporters.

>> This is your problem to fix.

> No, I've said from the start that this needs to be a community effort. A filesystem is too big for one person.

Right now that "community effort" is looking a bit unlikely, eh ?

I would hate to have to deal with these people as my primary occupation and I totally get why you don't want to continue.

That's said, nobody else has the power, skill or inclination to make bcachefs that wonderful filesystem of the future for Linux - only you. That's what I meant by "this is your problem to fix".

I wish you the best of luck with the new DKMS direction. And I'll get on board and actually try it out soon :D

> Please please don't forget I want you to succeed - that's why I bunged nearly $800 your way in this endeavour - but I'm not the only person who thinks you come across as completely immune to critisicm, even when it's constructive and from your supporters.

You have to understand, I get an absolute _ton_ of "constructive criticism" from people who basically are assuming everything is going off the rails and are expecting massive, and unrealistic changes; for bandwidth reasons if nothing else. I have to stay focused on the code and getting it done.

> Right now that "community effort" is looking a bit unlikely, eh ?

Actually, the big surprise from the DKMS switch is just how much the community came together to make it happen.

This project has looked like a one man show for a long time, and the core of it probably always will be for the simple reason that there are precious few people with the skillset required to do core filesystem engineering; that requires a massive amount of dedication and investment of time to get good at, there's a hell of a learning curve.

But there's still a lot of people with the willingness and ability to help out in other areas. People have been since even before the DKMS switch, to be honest, it's just that a lot of it is boring invisible but extremely necessary QA work - and that stuff is work, and people have helped out a lot there.

You have to be involved in the community, in the IRC channel to see this stuff going on. It's really not just me.

And now with the DKMS switch, a lot more people jumped in and started helping, and that's how we were able to get every major distro supported before the 6.17 release. That happened _fast_, and only a small fraction of the work was mine, mostly I was just coordinating.

Honestly, looking back, I don't think I could have planned this better - the timing was perfect. We're nearly done with stabilization, so it was the right time to start focusing more on distro integration and building up those working relationships, and the DKMS migration was just the kick in the pants to make that happen. Now we're pretty well positioned to get bcachefs into distro installers perhaps six months out.

The community is real, and it's growing.

(I would still _fucking love_ to have more actual filesystem engineers though, heh).

> I _really_ wanted Linux to get a native, modern COW filesystem

Btrfs not good?

(Honest question.)

I've been using btrfs for maybe 10 years now? -- on a single Linux home NAS. I use it in a raid1c3 config (I used to do c2). raid1cN is mirroring with N copies. I have compression on. I use snapshots rarely.

I've had a few issues, but no data loss:

* Early versions of btrfs had an issue where you'd run out of metadata space (if I recall). You had to rebalance and sometimes add some temporary space do that.

* One of my filesystems wasn't optimally aligned because btrfs didn't do that automatically (or something like that -- this was a long time ago.) A very very minor issue.

* Corruption (but no data loss, so I'm not sure it's corruption per se...) during a device replacement.

This last one caused no data loss, but a lot of error messages. I started a logical device removal, removed the device physically, rebooted, and then accidentally readded the physical device while it was still removing it logically. It was not happy. I physically removed the device again, finished the logical remove, and did a scrub and the fsck equivalent. No errors.

I think that's a testament to its resiliency, but also a testament how you can shoot yourself in the foot.

I've never used RAID5/6 on btrfs and don't plan to -- partly because of the scary words around it, but I also assume the rebuild time is longer.

Funny to hear your success; I've managed to break almost every mirror I've entrusted to BTRFS! How? Holding down the power button!

Seemingly regardless of the drives, interface, or kernel, other filesystems paired with LVM or mdraid fail/recover/lie more gracefully. NVMe or SATA (spindles). Demonstrated back-to-back with replacements from different batches.

Truly disheartening, I want BTRFS. I would like to dedicate some time to this, but, well, time remains of the essence. I'm hoping it's something boring like my luck with boards/storage controllers, /shrug.

Well, what are you waiting for? Get your findings to the btrfs-devel mailing list, include your drive make and model. Even better if it's reproducable.

TLDR: busy, lazy, not properly incentivized. I'll get right on that, boss... or is it Officer? I already said: time. I'd like to spend more of my time with triage before I disrupt others, particularly the developers. I don't mind y'all so much :)

It's reproducible, the scope needs to be reduced. With work. A lot of testing and variable change/reduction. More than I care for.

The problem: R&R, work/money, etc, all compete for a limited amount of time. I'll spend it how I like, Square? Comments win over rigorous testing with my schedule, thanks.

Why don't you try to reproduce it? Better things to do, this isn't the mailing list? Exactly. Pick a reason, there's plenty.

btrfs is good, but it's far from perfect. RAID 5 and 6 don't exactly work, it can have problems at high snapshot counts, and there's lots of even recent reports of corruption and other kinds of filesystem damage.

It feels more user friendly than ZFS, but ZFS is much more feature complete. I used to use btrfs for all my personal stuff, but honestly ext4 is just easier.

Reply on news.ycombinator.com