2,109 karma · joined July 30, 2012
If you write proprietary code, everything you do dies with that company. I certainly don't want my life's work locked away like that. Working on OSS means a better chance to put the engineering first and do something that will last.
I did my few years and Silicon Valley too, and when it came to decide between money and code, I chose the code. Haven't regretted a thing.
LSM-trees do really badly at multithreaded update workloads, and compaction overhead is really problematic when there isn't much update locality.
On the other hand, having most of your index be constant lets you use better data structures. Binary search is really bad.
For pure in memory indexes, according to the numbers I've seen it's actually really hard to beat a pure (heavily optimized) b-tree; for in-memory you use a much smaller node size than on disk (I've seen 64 bytes, I'd try 256 if I was writing one).
For on disk, you need to use a bigger node size, and then binary search is a problem. And 4k-8k as is still commonly used is much too small; you can do a lockless or mostly lockless in-memory b-tree, but not if it's persistent, so locking overhead, cache lookups, all become painful for persistent b-trees at smaller node sizes, not to mention access time on cache miss.
So the reason bcachefs's (and bcache's) btree is so fast is that we use much bigger nodes, and we're actually a hybrid compacting data structure. So we get the benefits of LSM-trees (better data structures to avoid binary search for most of a lookup) without the downsides, and having the individual nodes be (small, simple) compacting data structures is what makes big btree nodes (avoiding locking overhead, access time on node traversal) practical.
B-epsilon btrees are dumb, that's just taking the downsides of both - updating interior nodes in fastpaths kills multithreaded performance.
(obligatory plug for people to jump in and get involved with writing code if they want to see more of this stuff happen)
If you look at the field of filesystem developers and kernel developers, we don't have nearly as many young people getting involved and learning this stuff as we used to, and that's a problem. We need a pipeline of people building deep expertise, and if even a tiny fraction of the people getting involved with the bcachefs community start developing an interest and learning this stuff, that's a success.
Six months ago was also still a very hectic time for debugging and stabilization, it's definitely gotten better.
ZFS did a bunch of stuff right, it's just a much older design; pre-extents, and based on the original Unix filesystem design - filesystem as a database was still unproven at the time.
They were just working incrementally, which for the amount of new features ZFS already had was the smart decision at the time.
If we can, I'd like to get a quirks list in place, but there have been higher priorities.
Just after reconcile, I landed a patch series to automatically run recovery passes in the background if they (and all dependents) can be run online; this allows the 1.33 upgrade to run in the background.
And with DKMS, users aren't having to run old versions (forcing a downgrade) if they have to boot into an old kernel. That was a big support issue in the past, users would have to run old unsupported versions because of other kernel bugs (amdgpu being the most common offender).
As long as the testing pipelines are in place to make sure the dkms module builds on every distro configuration (a good chunk of that is still manual, but there's a project to improve the test infrastructure) - in practice, no one will notice.
I wouldn't have noticed the DKMS switch on my NixOS laptop if I didn't know it was happening.
This is hacker news, not drama queen news :)
related, plymouth integration to let users know when their machine is booting up if a drive or the filesystem is unhealthy
We're not aiming to be in GUI installers yet, that'll be sometime after taking the experimental label off. We're still going slow and steady; I don't think about doing things that will bring in more users until incoming bug reports are dead quiet (or as close to it as they ever get), and the userbase has been going up plenty fast all on its own by the activity I see.
So, sometime next year we'll be working on distro stuff again. Dunno when, I expect another spike in new users and bug reports when I take the experimental label off.
While reconcile was getting done we got a detailed outline of where EC resilvering is going to plug into that, so it's not looking like a huge amount of work anymore - and there's been people testing EC and reporting the occasional bug, it's been looking pretty solid.
We did some performance testing not too long ago, and it looked like we were in better shape than I thought. I'm still more interested in tracking down performance bug than shaving cycles and going for raw IOPS.
And the userbase isn't complaining about performance at all, aside from the odd thing like accounting read being slow (just fixed a couple issues there) or lack of defrag.
After debugging and stabilization, it's going to be more about usability, fleshing out missing features, more integration work (there's some systemd integration that needs to happen in the mount path), telemetry/introspection improvements (I want all the data I can get for stabilization, and json reporting would be good for lots of things).
So, if you're asking if you can help, that's a decent list to start from :)
We've been cranking through bugs fast, and there are still bug reports coming in but the severity and frequency has declined drastically, while the userbase has gone up; polling the userbase it's been stabilizing fast.
But we won't really know we're there until we're there, so the main thing I can say is: if you report a bug like that, it'll get looked at fast; the debugging tools are top notch.
Happy to answer questions about all things bcachefs or what-have-you.
just please no more questions about whether or not bcachefs will be in the kernel, I've been asked that enough :)
Boomers coasted off the success of the postwar era, and now we're all poorer for it.
It'll go faster once all the bindings are in place and people have more experience with this stuff. I've been greatly looking forward to expanding bcachefs's use of rust, right now it's just in userspace but I've got some initial bindings for bcachefs's core btree API.
Real iterators, closures, better data types, all that stuff is going to be so nice when it can replace pages and pages of macro madness.
You have to understand, I get an absolute _ton_ of "constructive criticism" from people who basically are assuming everything is going off the rails and are expecting massive, and unrealistic changes; for bandwidth reasons if nothing else. I have to stay focused on the code and getting it done.
> Right now that "community effort" is looking a bit unlikely, eh ?
Actually, the big surprise from the DKMS switch is just how much the community came together to make it happen.
This project has looked like a one man show for a long time, and the core of it probably always will be for the simple reason that there are precious few people with the skillset required to do core filesystem engineering; that requires a massive amount of dedication and investment of time to get good at, there's a hell of a learning curve.
But there's still a lot of people with the willingness and ability to help out in other areas. People have been since even before the DKMS switch, to be honest, it's just that a lot of it is boring invisible but extremely necessary QA work - and that stuff is work, and people have helped out a lot there.
You have to be involved in the community, in the IRC channel to see this stuff going on. It's really not just me.
And now with the DKMS switch, a lot more people jumped in and started helping, and that's how we were able to get every major distro supported before the 6.17 release. That happened _fast_, and only a small fraction of the work was mine, mostly I was just coordinating.
Honestly, looking back, I don't think I could have planned this better - the timing was perfect. We're nearly done with stabilization, so it was the right time to start focusing more on distro integration and building up those working relationships, and the DKMS migration was just the kick in the pants to make that happen. Now we're pretty well positioned to get bcachefs into distro installers perhaps six months out.
The community is real, and it's growing.
(I would still _fucking love_ to have more actual filesystem engineers though, heh).
I can't ship and support a filesystem under those circumstances.
The "support" angle is one you and a lot of other people are missing. Supporting it is critical to stabilizing, and we can't close the loop with users if we can't ship bugfixes.
Given the past history with btrfs, this is of primary concern.
You've been looking for a compromise position, and that's understandable, but sometimes the only reasonable compromise is "less stupidity, or we go our separate ways".
The breaking point was, in the private maintainer thread, a page and a half rant from Linus on how he doesn't trust my judgement, and then immediately after that another rant on how - supposedly - everyone in the kernel community hates me and wants me gone.
Not joking. You can go over everything I've ever said on LKML, including the CoC incident, and nothing rises to that level. I really just want to be done with that sort of toxicity.
I know you and a lot of other people wanted bcachefs to be upstream, and it's not an ideal situation, but there are limits to what I'm willing to put up with.
Now, we just need to be looking forward. The DKMS transition has been going smoothly, more people are getting involved, everyone's fully committed to making this work and I still have my funding.
It's going to be ok, we don't need to have everything riding on making it work with the kernel community.
And now, you guys don't have to worry about me burning out on the kernel community and losing my last vestiges of sanity and going off to live in the mountains and herd goats :)
Citation needed.
> Linus's job is not to make the very next version of the kernel as good as it can be. It's to keep the whole system of Linux kernel maintenance going. (Maintaining the quality of the next version is almost a side-effect.) Asking him to make frequent exceptions to process is the equivalent of going "this filesystem is making poor decisions
You're arguing from a false premise here. No exceptions were needed or required, bcachefs was being singled out because he and the other maintainers involved had no real interest in the end goal of getting a stable, reliable, trustworthy modern filesystem.
The discussions, public and private - just like you're doing here - always managed to veer away from engineering concerns; people were more concerned with politics, "playing the game", and - I'm not joking here - undermining me as maintainer; demanding for someone else to take over.
Basic stuff like QA procedure and how we prioritize patches never entered into it, even as I repeatedly laid that stuff out.
> > If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.
> You have missed the point by a mile.
No, that is very much the point here. bcachefs has always had one of the better track records at avoiding regressions and quickly handling them when they do get through, and was being singled out as if something was going horribly awry anyways. That needs an explanation, but one was never given.
Look, from the way you've been arguing things - have you been getting your background from youtube commentators? You have a pretty one sided take, and you're pushing that point of view really hard when talking to the person who's actually been in the middle of it for the past two years.
Maybe you should reevaluate that.
Maybe if you're a distance observer who isn't otherwise participating in the project
What's the saying about too many cooks in the kitchen?
These concerns aren't coming from the people who are actually using it. They're coming from people who are accustomed to the old ways of doing things and have no idea what working closely with modern test infrastructure is like.
There wasn't anything unusual about the out-of-merge-window patches I was sending Linus except for volume, which is exactly what you'd expect in a rapidly stabilizing filesystem. If anything I've been more conservative about what I send that other subsystems.
> It certainly appears that Linus can't tell the difference between your new feature introductions and your maintenance fixes, and that should trigger some self-reflection. He clearly couldn't let all of the thousands of different kernel components operate in the style you seem to prefer for bacachefs.
If Linus can't figure out which subsystems have QA problems and which don't, that's his problem, not mine.