Orphaning bcachefs-tools in Debian
jonathancarter.org
jonathancarter.org
It feels like whenever the author of bcachefs comes up, it's always because of some drama.
Just the other day he clashed with Linus Torvalds: https://lore.kernel.org/lkml/CAHk-=wj1Oo9-g-yuwWuHQZU8v=VAsB...
My reading is that he's very passionate, so he wants to "move fast and break things" and doesn't get why the others aren't necessarily very happy about it.
Edit: Reading through the thread, it seems like there is a claim of rigorous but effectively private testing. Without the ability to audit those results easily it’s causing a lot of worry.
Linus' complaint was about adding feature work into a supposed minor bug fix, especially because (going from Linus' comments) they were essentially refactors of major systems that impacted other areas of the kernel.
If there is a defect found, the change to a stable branch should literally be the minimal code change to fix the reported issue. Ideally, if it's a newly introduced issue (i.e. since being on the stable branch), the problematic code reverted and a different fix to the original defect applied instead (or left if it's deemed less of an issue than taking another speculative fix). Anything that requires a re-organisation of code, by definition, isn't a minimal fix. Maybe it's the correct long-term solution, but that can be done on the unstable branch, but for the stable branch, the best fix is the simplest work around. If there isn't a simple work around, the best fix is to revert everything back to the previous stable version and keep iterating on the unstable branch.
The guy even admits it as well with his repeated "please don't actually use this in production" style messages - it's hard to give a greater indication than this that the code isn't yet ready for stable.
I can understand why from his perspective he wants his changes in the hands of users as soon as possible - it's something he's poured his heart and soul and he strongly believes it will improve his users' experience. It's also the case that he is happy running the very latest and probably has more confidence in it that an older version. The rational choice from his perspective is to always use the latest code. But, discounting the extremely unlikely situation that his code is entirely bug free, that just means he hasn't yet found the next serious bug. If a big code change is rushed out into the stable branch, it just increases the likelihood that any serious bug won't have the time it needs in testing to have the confidence that's the branch is suitable for promotion to release.
Considering that Kent himself warns against general use right now, I don't quite see the urgency to get the bug fixes out - in my understanding Linus would happily merge them in the next development kernel. And whoever is set to to run bcachefs right now, might also be happy to run a dev kernel.
If the bug it's fixing was already in the current release branch, wasn't noticed before and has only shown up now late in the stable branch lifetime, then it definitely doesn't seem like something that needs an urgent fix.
Whether this internal process gives him a pass for getting his non-regression fixes in after the merge window is at the end of the day for Linus to decide. And Linus is finally erring on the side of "Please just do what everyone else is doing" rather than "Okay, fine Kent, just this once".
I would say it's ironic to start a comment saying: "It's very clear from that thread that he doesn't understand the purpose of the stable branch" when it's "very clear" from your opening paragraph that you don't understand the thread.
Taken at it's most charitable, the opening of the first message "no more bug reports related to the disk accounting rewrite, things are looking good over here as far as regressions go" would suggest a meaning of "there are no significant bugs, the changes below are optional".
The next section in the change description then says that this fixes a number of very serious bugs. Straight away, I can see the potential for an interpretation difference. Is it "heads up, no changes required" or "these fixes are critical"?
He's told "no" by Linus, for reasons that seem to correlate with what I said (unless you'd like to point out in what way I don't understand the thread), and then rather than saying "yeah, then can wait until the next stable branch", he doubled down on the importance of getting these changes in and basically saying that the rules should only apply to everyone else and not him because he knows that there won't be any new bugs because of $REASONS. $REASONS that didn't apply when the bugs were introduced. $REASONS that include automated testing, but that didn't find these bugs originally.
The thread (which apparently I don't understand) contains a perfect summary from Linus himself: "But it doesn't even change the issue: you aren't fixing a regression, you are doing new development to fix some old probl;em, and now you are literally editing non-bcachefs files too."
All this for some changes to a system that he's actively discouraging people from using because it's not production ready anyway, and so none of these bug fixes are actually critical for right now.
It's good he ultimately backs down, but he should never have been pushing for these changes this late in the stable branch timeline anyway.
So, that's my understanding of the thread. I'd be interested to hear how your understanding of the thread is so radically different from that.
Fundamentally on the whole I don't think most of your interpretation is comment worthy. (To clarify, I don't think its particularly objectionable following from the premise in your opening paragraph.) But...
> in the stable branch timeline
Again. Like I outlined in my initial reply. This has nothing to do with stable. I don't know why you keep talking about stable.
The discussion is about bleeding edge mainline Linux. It's clear to me because:
* It is a PR for Linus. I don't know enough about stable to know if they use PRs but most stable stuff I do know about involves marking patches for stable on the specific mailing lists oriented around stable.
* Linus doesn't handle stable.
* Linus and Kent are taking about the merge window and Kent submitting non-regression fixing patches after it. This doesn't make any sense if it was in a stable context. The process is different.
* If this was stable the discussion would be with GKH.
So, your comment is based on the premise of this being a discussion surrounding stable. It's not, so I don't know what to make of the rest of your comment on the basis of this incorrect premise.
The repo is literally called "linux-stable-rc"
It wasn't an incorrect premise, just incorrect terminology. Sorry, my bad, I shouldn't have referred to it prematurely as "stable" when it is just undergoing the process of stabilization.
> > in the stable branch timeline
> This has nothing to do with stable. I don't know why you keep talking about stable.
Yes, technically, this isn't officially called "stable" until after the last release candidate, however every release candidate should be considered as an attempt to create the stable release (although pragmatically, nobody expects the first few to have had enough testing to surface all the bugs that are likely to show up) and I don't think it's particularly egregious to talk about this change in the context of the stable branch timeline as -rc releases are just as much part of the timeline as the initial stable release and the later point releases.
For context, this change was being requested for inclusion in -rc6, which was over 4 weeks after the merge window ended. This very well could end up being promoted to the stable release if no more significant bugs are found. There is no way a change of this complexity should have been accepted, and when Linus pointed that out, Kent shouldn't have been arguing about it at all, instead he should have just waited to get it merged into 6.12 as he originally intended.
> The discussion is about bleeding edge mainline Linux.
Yes, it's mainline, but also "bleeding edge" is kind of a misnomer, as it hadn't been accepting feature changes and was in stabilization for producing stable release candidates for a month already, and by that point would have had significant testing.
Sorry for causing confusion my referring to it prematurely as "stable". I don't look at the kernel all that often, and we use a different process with different terminology in our environment. We keep mainline open all the time for ongoing feature work, fork that to "stable" which only accepts bug-fixes and from that we periodically create release candidates which get released for testing and possibly get relabeled as the actual release. Sorry I was still thinking in that mindset when I replied and didn't properly map the concepts back to those used in the kernel.
It's not? What repo? The only two repos which are involved are https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... which is mainline (implicitly) which is where these patches would land and git://evilpiepirate.org/bcachefs.git which is the source repo for the PR. The only branches being referenced are "master" (implicitly) for mainline and the tag "bcachefs-2024-08-23".
Regardless, to respond to the rest of your comment:
The reasons for why Linus is rejecting the change have nothing to do with the stable process and everything to do with the set release process. The mainline merge window opens, you (not you specifically unless you are a subsystem maintainer, if you want to contribute a patch as a non-maintainer, the process is completely separate and goes via the subsystem maintainers) submit features and bug fixes, the merge window closes, somewhere in the ballpark of 7 release candidates happen, and it's released as mainline. The goal of the RCs is to incorporate subsequent waves of fixes for any regressions introduced specifically by the bug fixes and new features.
Kent is claiming that, because he himself implements effectively an equivalently rigorous (according to him) feature testing and stabilisation process that his patches which do not fix regressions introduced by previous patches submitted during the merge window, but which do fix some real bugs, should be accepted outside the merge window.
In the past, Linus has let it slide, and he has also let it slide this time too: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... . Linus is just asking Kent to stop doing this as he doesn't want to keep giving him special treatment.
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
There are repos on git.kernel.org for microemacs, that doesn't mean this thread relates to microemacs.
Upstream stable kernel certainly does not care about compatibility with your particular thirdparty module. You'll just have to add another KERNEL_VERSION #if. Maybe if you're nvidia, or something, things are different.
ZFS has broken on new releases (I don't recall if they were stable, I think they were), and that is one reason I won't use as the main filesystem on linux.
True that, and yet the kernel has zero issues keeping Btrfs around even though it's been eating people data since 2010. Kent Overstreet sure is naive at times, but I just can't not sneer at the irony that an experimental filesystem is arguably better than a 15-years old one that's been in the Linux kernel for more than a decade.
I can imagine scenarios where known failure modes on an "inferior" tool are better than unknown failure modes on a "superior" one.
I thought Carl Thompson's response was very good and constructive: https://lore.kernel.org/lkml/1816164937.417.1724473375169@ma...
What I don't understand is that IIUC Kent has his development git history well broken up into small tight commits. But he seems to be sending the Linux maintainers patches that are much larger than they want. I don't get why he doesn't take the feedback and work with them to send smaller patches.
EDIT: The culture at Google (where Kent used to work) was small patches, although that did vary by team. At Google you have fleet-wide control and can roll back changes that looked good in testing but worked out poorly in production. You can't do that across all organizations or people who have installed bcachefs. Carl pointed out that Kent seemed to be missing some social aspects, but I feel like he's also not fully appreciating the technical aspects behind why the process is the way it is.
I included the rcu_pending and vfs inode rhashtable conversion because I was getting user reports that it fixed issues that were seriously affecting system usability, and because they were algorithmically simple and well tested.
Back in the day, on multiple occasions Linus and others were rewriting core mm code in RC kernels; bcachefs is still experimental, so stuff like this should still be somewhat expected.
Hope you don't get too much into trouble with Linus. I do not want to see you or the project get into the wrong side of the old guard...
I really think you need to realign your expectations here. The Linux kernel is in a different place now than "back in the day" and you are not Linus Torvalds.
That PR would have been better off had it been split into multiple ones and timed differently.
I think that dubious distinction would go to Hans Reiser.
That's not how I read that thread. This is just about where the diligence happens, not that it can be avoided; and exactly how small a fix is mergable in the fixes phase of kernel development.
I don't see that thread as being particularly angry either. There have been ones where both of them have definitely lost their cool; here they are having a (for them) calm disagreement. Linus is just not going to merge this one until the next development phase, which is fine.
There have been arguments involving this developer that do raise questions; I just don't see this as one of them.
This is just about following procedure. There are people who follow procedure and introduce many bugs, and people who don't but write perfect software.
The bcachefs author cautiously marks bcachefs as unstable, which is normal for file systems. The only issue here is that the patch touched other areas, but in the kernel development model Linus is free not to pull it.
In this industry it's very, very easy to run in circle, keeping doing stuff over stuff without realising you are in a bikeshedding loop, you're overengineering or simply wasting time. We need people that want to just push forward and assume the responsibility of anything that breaks, otherwise I'm sure that in 30 years we'd all still be using the same stuff we've always used because it's just human nature to stick with what we know works, quirks and all.
The issue raised is that some program written in rust insists on a very specific version of various dependencies. If other people change the metadata, it builds and seems to run ok with different versions. (Developer on reddit clarifies that it builds and does the wrong thing, and recommends dropping Debian as a solution).
Linux likes to pack a finite set of versions of libraries (ideally just one version at a time) and use that dependency for multiple programs, totally at odds with the vendoring specific versions strategy.
I'm not clear what a solution is to this one. In the specific case, sure, drop it from Debian. But this isn't the only program choosing to vendor dependencies instead of use them from the environment.
But that same issue affects Ruby, Python, etc. if the project specifies strict versions. And if you deal with filesystems, you really want strict versions - I understand why the author wants to keep it that way.
It's more of a self-inflicted Debian policy pain rather than anything to do with Rust. The author could be nice and vendor, but also he's completely right to refuse. The request is basically "we put handcuffs on ourselves, so please you do the work instead".
if you deal with filesystems, you really want strict versions
This is a bad dependency management, nothing more. Vendoring is a PITA.
It's similar to why restic (backup software) has official, static, reproducible binaries in their releases.
And if whatever library bugged, oh well, random people can access my systems ?
And if my bgpd bugged, oh well, we just inject random routes et forward data randomly ?
Makes no sense.
Relaxing library requirements can introduce security issues when the dependency behaviour changes, so I'm not sure what the argument is.
If your bgp is bugged, yeah it sucks, but it's recoverable. We're using encrypted traffic these days to ensure that even captured traffic is safe.
The difference here is that data corruption is not recoverable.
RCE would have been worse than file system bugs, as it leads directly to a superset of problems including potentially losing all your files.
C/C++ built with Bazel or Conan will have the same issue too.
1. Why Rust Libraries May Never Exist - https://www.youtube.com/watch?v=769VqNup21Q
2. The Rust Cargo ecosystem is broken - https://www.youtube.com/watch?v=OdMQwDtumJ8
As much as its standard library superficially looks close to C, it is as equally unlike C in its performance and abstraction level. It does not have suitable tools to do systems programming properly and its compiler and interop capabilities are too weak.
Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obviously want these "containers" to use your real hardware. And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.
This is a non-issue.
libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all.
Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.
Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.
Yes, yes it will. The baseline stuff (the base protocol and xdg_shell) has been stable for years. Of course, one might argue that the caveman byte poking provided there turns a modern machine into a slightly faster 15-year-old one, and I agree it does. (It’s still fun to do as an experiment.)
But the moment you try to actually leverage the GPU the modern Linux desktop forces you to abandon static linking in order to load pieces of the graphics driver into your address space, and unfortunately those pieces will in turn force you to dynamically link the distro’s preferred libc (likely Glibc) and libwayland (as opposed to any other protocol implementation).
So either you poke pixels into a byte buffer by hand, and your question gets an answer in the affirmative simply because that foundational layer is fairly small, or you take advantage of the hardware, and your question becomes moot because you can no longer statically link.
This is what I call "version soup".
The idea that every project can choose an near-arbitrary set of very exact version numbers of dependencies and expect it to work. And that every project on earth has the burden of continually stirring their soup, often through a tool like dependabot, hoping (cross fingers) no compatibility problems come about.
Of course, there's no guarantee that those versions will work together. The extremely limited abilities packaging tools have for expressing dependency restrictions (>= this or that version) generally lack the ability to even handle the concept of stable branches with backports. i.e. "No, it must be >= 1.2.3.. what do you mean they backported the relevant fix to 1.1.4? 1.1.4 < 1.2.3 so it's unacceptable". The way these specifications are then used by authors brings an extra layer of noise to the compatibility situation - very few (understandably) will actually go and check the limits of version compatibility.
In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports.
Is this different from any other Linux distribution?
- most Linux distributions don't reach very far into e.g. the python ecosystem (python packages that exist are generally there to support packaged applications)
- nixpkgs isn't linux specific - most packages work on macos, and nixpkgs happily works on top of other distributions.
So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.
The same applies to Go.
It creates a security nightmare.
Distributions expect to be able to security patch without waiting for laggard upstreams to bump their pinned dependency versions. If the ecosystem insists that only upstream-sanctioned versions are acceptable, then that runs contrary to this expectation, because it blocks distributions from patching or bumping dependency versions. If this case should be an exception, then you end up with a security team having to patch myriad different versions. That's the security nightmare.
I do think we need to reevaluate how security is managed in terms of software. It's often the case that we say "this library version has a CVE, we need to update it!" However, it's not often the case that the CVE in the library is even exploitable from the parent application.
I'm thinking of a recent 9 CVE against zlib... But actually it was a rarely used extension API for working with compressed files.
Something like path tracing to see if a vulnerable API is even accessible from the parent application/library before instructing the universe to adopt a new version.
You could imagine a model where security vulnerability reports include a combination of features required to trigger that vulnerability. If the vulnerability was in a rarely-used extension, like it was for Zlib, many dependents would be automatically marked as non-vulnerable because they wouldn't have the necessary feature flag active.
Not to mention, this already happens to some extent, especially for older versions of software: distributors and developers do sometimes say "we will not provide a patch because even if we use the affected library, we don't use it in a way that triggers the vulnerability".
Take Nixpkgs, for example. They have this concept called "default crate overrides" which their rust builder uses, which offers the capability to patch all versions of a crate. That's not to say that Nixpkgs does rust perfectly, but just an example of the sort of tooling a security team should have at their disposal.
FUD enuses.
You can also blend both strategies - you might have a version minimization strategy, where you only use, for example, one major version of a library at a time.
There are also more tools than ever to do things like correlate versions of OSS packages with reported vulnerabilities e.g. Google's osv.dev. 95% or more of Rust vendoring isn't actually "vendoring" in the sense you have to figure anything out. The dependencies are literally written in the lock file with the version and that can be traced back to a Git hash automatically in almost all cases. Some crates that bind C libraries might require this work, where you have to look at the code to see "What version of the code did the `cp -R` into here?" but they are practically limited in number, and detectable. You aren't doing any detective work for 98% of this stuff. It can be scaled pretty well with a few (good) heuristics.
Rust even has the advantage that large, vast amounts of code is actually statically analyzable under known sets of configurations/features. In the future I suspect a tool like `govulncheck` for Rust will be developed, making accurate vuln tracing even easier.
While I'm admittedly biased because I work on a distro where this problem isn't as directly relevant, my answer is that in 2024 if you don't have the tooling to do this stuff, it's absolutely on you. Whether as a policy decision or a technical limitation, whatever.
As the sibling comment says, the issue is when the patch doesn't apply cleanly to some subset of versions in use, or even worse if they appear to apply cleanly but leave behind logic bugs.
If distributions did approach this by "distro-patching" specific versions, then that would reintroduce the upstream complaint that distros are using versions of dependencies not sanctioned by them. So your approach, even if it did work in the general case, would take you back to square one.
But most importantly, I don't see any argument this is all meaningfully more work than all the manual shit Debian maintainers do to get all this stuff working, on top of the fact they try to ship frankenstein versions of packages that silently break and piss off users (every distro ships some busted packages but it's a matter of give and take.) Other maintainers of other distros don't spend time on all the stuff the OP spent time on. They don't spend time loosening lockfile bounds on Rust crates for filesystem-sensitive tooling(!!!) in an effort to make a 35-year-old cultural rule about C/C++'s compilation model apply to them. Mostly, they run some script like 'import crates.io package XYZ version 1.2.3', commit the result, and get it code reviewed and merged. They can then spend time on other things. And often, these same processes are used for all updates, including many security updates. This stuff is insanely well oiled in other projects.
Sorry to be the hater, but this is squarely a Debian dysfunction as far as I can see? If they spent more time actually solving infrastructural problems they had, they would be able to spend time on actually delivering more working software to their users in the world they actually exist in.
> this is squarely a Debian dysfunction as far as I can see
Nothing I have said is specific to Debian or to C/C++. I've presented general arguments that hold regardless of distribution policies or the language toolchain used. Nothing you have said demonstrates why the general points I have made would apply to Debian or to C/C++ but not to Rust-based projects.
If you still believe that this is the case, then I think that demonstrates that you do not understand the matter deeply enough.
The only thing that Debian is insisting on is basic software supply chain hygiene. Every so often, there is a very public failure in ecosystems that don't bother with this hygiene, while Debian continues unaffected.
> You can apply patches to most versions of a crate
The point of the complaint that spawned this discussion is that there's an upstream that considers it unacceptable for downstreams to be doing this patching (or indeed running any dependency version that is not sanctioned by upstream). It is a contradictory position to justify downstream patches as a supposedly easy solution while at the same time complaining that downstreams patch at all.
The fact is that distributions users do not expect to be beholden to laggard upstreams while awaiting security fixes. This requires distributions to modify the dependency versions (or patch, which is the equivalent "off-piste" behaviour) used to build their packages.
Almost. You can write dynamic libraries in Rust (using the cdylib target type), but their external interface will be limited to the C ABI. For all intents and purposes, they'll be 'C shared libraries' that just happen to be written in Rust. (You can also create C headers for the exported C API using cbindgen.)
That's not true.
$ ldd /usr/bin/rustc
linux-vdso.so.1 (0x00007fde30c44000)
librustc_driver-adb68da129cf8bba.so => /lib64/librustc_driver-adb68da129cf8bba.so (0x00007fde2cc00000)
libstd-aa46ce58aa106bb2.so => /lib64/libstd-aa46ce58aa106bb2.so (0x00007fde2cad3000)
libc.so.6 => /lib64/libc.so.6 (0x00007fde2c8e2000)
[...]
What Rust currently doesn't support (there's a desire to fix this someday), is dynamically linking with a Rust library compiled with a different version of the Rust compiler, unless the library has been restricted to a C-compatible ABI. That's because Rust currently does not have a stable ABI; things like structure layouts can change between versions of the compiler, and Rust makes heavy use of inlining.
Note that even with the same compiler there a lot of factors that must remain the same to make the dynamic linking work properly. For example changing any compiler flag or dependency feature flag will likely result in incompatibility. The best way to guarantee that the dynamic library and the executable that loads it are compatible is to build everything as part of a single workspace (and yes, this means that if anything changes in the dynamic library crate then the executable that depend on it must also be rebuilt).
Vendoring used to be more common, even in Debian. But then there was an important zlib vulnerability (which IIRC could be exploited through manipulated compressed data), and they had to chase and fix all the many copies of zlib embedded all over the place. To make things worse, some of them were not only old versions of zlib, but also had modified that zlib code, so each had to be reviewed before applying the fix.
Debian (and other traditional distributions) learned from that incident, and started a strong push to not only remove vendored copies of libraries, but also when possible use a single version of these libraries. Which means that, whenever an important security issue is once again found in a widely used library, only a single copy has to be fixed and updated.
[0]: https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_...
Looking at the debian-devel archives around that date, I found a Debian developer complaining about the vendored zlib copies (https://lists.debian.org/debian-devel/2002/03/msg00716.html), but not the full discussion about getting rid of vendored libraries, so it must have happened elsewhere.
Can confirm, I occasionally use an ai program that has to hard code all of their python dependencies, because it's the only way to get it to compile, let alone run properly... and then they go and change the underlying package manager, and figure out they have to hard code even more... it's a bloody mess, but thankfully you can just run it as a docker container once it's all working...
Version locking is a powerful tool that should be used very sparingly. My 2c.
For Rust it is language specific, because the thing being statically built, doing differently is very hard.
For other languages like Python, there is not a need and you should not do that. The dynamic part of it make it so that developer should be resilient to different versions. And python also provide a lot of conveniences for that (just think about six).
But in your case, you faced the current situation of lot of "data scientist" jumping the band wagon of AI are creating a lot of things like apprentice sorcerers without a real experience of software engineering.
Normally I would agree, but when it comes to cuda and rocm, and the one that intel uses... you find that you have to pin torch/audio/video, and then you have to start pinning its dependencies (and their dependencies etc etc) or else you start getting random build or runtime failures...
It can be pretty bad with the bigger ML/scientific computing packages, hence all the version pinning that happens, though I will concur that generally speaking AI researchers don't understand package management in the slightest and you get all kinds of horrors as a result (e.g. mixing anaconda and pip in the same project, bonus points if it's blasted over the distro's python packaging for even more chaos. works on one very broken machine if you are lucky and don't touch anything).
Debian's approach is similarly frustrating: they will try to make your code work with an older version of a library than the one you developed with, which doesn't even work within semver (semver only defines backwards compatibility, not forwards). That's what caused the problem in this case: rolling forward dependencies was not recommended but didn't break anything, rolling a dependency backwards meant it stopped working, which is something absolutely no-one should be surprised or upset by (at least it meant the build broke as opposed to debian shipping a broken package like would have likely happened with python. It just meant that debian was shipping a version with bugs fixed in upstream). If you're going to do that you've got to be prepared to take responsibility for fixing the problem that you've introduced.
Debian is a great OS, but it targets stability and long term support for its releases. That just isn’t compatible with newer, faster moving software that’s still working towards stability. I remember it being an issue when I was playing around with Mono around 2010 ish, and it’s an issue now with bcachefs - a very new and fast moving technology.
For motivated users, the solution is to install bcachefs-tools direct from upstream, or from a third party packager (if one exists). When bcachefs stabilises, I’m sure it’ll find its way back into Debian official.
That said, having dabled in rust there’s a temptation to name breaking changes to improve ergonomics because the type system is so expressive. I almost wonder if this is an argument for simpler type systems to reduce churn.
My point was "some of the upstream versions were older than Debian's" which means the "Debian is behind, Debian is slow" is not true in the case. That usually only happens where an upstream is either very conservative with bumping versions, or where upstream is under-maintained, neither which at first glance seem to be true of becache-fs.
(There is a somewhat frustrating trend in rust packages for them to sit on 0.x versions for a very long time, even if they're fairly heavily used in the ecosystem and pretty stable in practice)
From what the author of bcachefs is saying, the forward moves were not a problem, it was moving bindgen backwards which broke the build, meaning the packaged version of bcachefs-tools stayed very out of date, causing him to get bug reports for known and fixed fairly serious bugs (ones which rendered machines unbootable).
Doing this has a couple of key advantages:
1. when a security vulnerability needs patching in a stable distribution release, the security team only have to patch one version instead of a dozen or more different versions bundled all over the place
2. a library that depends on dependencies A and B, both of which depend on X, can actually work properly if it needs to pass API objects created by X across between A and B, since X is of the same version
In an ecosystem where it's considered acceptable to "pin" versions of dependencies and also call any system that doesn't use the pinned versions "unsupported", both of the above two cases become impractical.
Whether you use shared libraries or static libraries, the above matters still exist.
That feels like an advantage for the developer, not the distro maintainer.
I find that amusing, because you're saying it's easy if they can use one version of the library. Well, that's also true if they're writing an application, hence pinning.
It's an advantage for the ecosystem, yes. Not the developer of the library (X in my example), but a different developer. Someone who is trying to write an app to do something interesting with A and B.
> ...because you're saying it's easy if they can use one version of the library. Well, that's also true if they're writing an application
I don't think that's true.
Perhaps library A has pinned dependencies for X such that it is in conflict with unrelated library B that also uses pinned dependencies for X. Both libraries are now less useful to the ecosystem, because they are now inadvertently incompatible with each other (if their APIs expose X's objects).
My example demonstrates why this doesn't work in the general case, and therefore why it's wrong to expect that everyone does it this way.
1. Rust is as hostile to dynamic libraries as glibc to static.
2. With everything static you have to rebuild every dependent on any security patch anyways. If you meant with multiple versions maintainers have to backport patches to multiple versions. Maybe don't backport at all? Some people appreciate having backports. Users of software written in Rust does NOT. So why bother doing backports for them?
3.
> a library that depends on dependencies A and B, both of which depend on X, can actually work properly
This will be detected during compile (since it's all static! sorry dynamic linking fans) and dealt with.
For statically compiled apps, if the version is over specified recompiling will have no effect. The dependent will have to be updated with a new version string.
If it is correctly detected, then the only "dealing with" that can be done is to fail to build.
This is completely false. First and foremost, people who use applications don't care at all what language those apps are written in, and there is no "community of people who use Rust tools" that could have a different world-view from everyone else.
And beyond that, there is no one in the world who doesn't want backports, because it's never safe to just take the latest version of some piece of software and expect that everything will still work as expected. Having to take even a month of new features just because you need a security fix.
And note that Rust even at the language level has strong support for backwards compatibility and support for older versions. It's definitely not true that Rust people, even developers, live at the bleeding edge and don't need backports.
If tests falsely pass, then that's a quality problem in the upstream tests.
To deal with this today you install a stable base system with an overlaid "user" package system so Debian/Fedora + nixpkgs/guix/brew/snap/flatpak/mpr/pip/cargo/etc. Unfortunately because because although you can get most packages on nixpkgs you can't get all and you'll need to install multiple so it becomes a nightmare to maintain (how do you remember all the package managers you have to update during the xz vulnerability?) and extremely bloated due to duplicated dependencies (especially for GUI packages that require gnome or kde).
You do get pretty far for CLI-only packages by just adding nixpkgs though. Too bad it's so terrible to use.
MPR is also pretty interesting: You basically leverage the AUR for Debian and with a few changes you could probably make the dependency names translation automatic and transparent. It solves the bloat problem (since you'll mostly use system packages) but doesn't help with library versions.
Though I agree that it is closer to a system-critical package than most packages.
Why can't Debian do that though?
Moreover vendoring is generally considered a bad practice, more so by Debian, so this just seems a step backwards.
Can you link to the post on Reddit you are referring to?
https://www.reddit.com/r/bcachefs/comments/1f4erbg/comment/l...
It only insists on semver-compatible versions (if a Rust/Cargo package specifies libfoo=5.1, it will work with libfoo=5.9). It's one per major version, not that different from Debian packaging "libfoo5" and "libfoo7" when both are needed.
The difference is that Cargo unifies versions by updating them to the newest compatible, and Debian unifies them by downgrading them to old unsupported versions, ignores compatibility, and reintroduces bugs, and disables required functionality.
Kent's reply in that thread has more details.
Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.
This is just all-around weird what Debian is doing.
(Full disclosure, I am the one who introduced the first piece of Rust code into bcachefs-tools)
When libraries are removed you have to update your build. This will not be the case if you use your own infrastructure (crates.io mirror).
In practise, this means that Debian has to make its own copy of the source code available from a Debian-controlled repository, to ensure that no (accidental or otherwise) change to an upstream source archive can cause non-DFSG compliant Debian source or binary packages to be distributed.
keeping a debian-hosted copy of all source used to build packages seems like a very reasonable defensive move to minimize external infrastructure dependencies and is key in the reproducible builds process
there's definitely a conflict with the modern style of volatile language-specific package management, and I don't think debian's approach is ideal, but there's a reason so many people use debian as a base system
also it seems like the idea of maintaining stable branches of software has fallen out of vogue in general
AFAIK, most traditional distributions do that, not just Debian. They consider it important that software can be rebuilt, even in the far future, with nothing more than a copy of the distribution's binary and source packages. Doing anything which depends on network access during a build of the software is verboten (and AFAIK the automated build hosts block the network to enforce that requirement).
Keep also in mind that these distributions are from before our current hyper-connected time; it was common for a computer to be offline most of the time, and only dial up to the Internet when necessary. You can still download full CD or DVD sets containing all of the Debian binaries and source code, and these should be enough to rebuild anything from that distribution, even on an air-gaped computer.
> Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.
That is often done for C libraries; for instance, Debian stable has both libncurses5 and libncurses6 packages. But it's a lot of work, since for technical reasons, each version has to be an independent package with a separate name, and at least for Debian, each new package has to be reviewed by the small ftpmaster team before being added to the distribution. I don't know whether there's anything Rust-specific that makes this harder (for C libraries, the filenames within the library packages are different, and the -dev packages with the headers conflict with each other so only one can be installed at a time).
There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).
This is the most important part. Debian LTS maintains packages for 5 years. Canonical takes Debian sources, and offers to maintain their LTS for 10 years. Red Hat also promises 10 years of support. They don't want anything in the core part of their stable branches that they can't promise to maintain for the next 5-10 years, when they have no assurance that upstream will even exist that long.
If you want to move fast and break things, that's also fine. Just build and distribute your own .deb or .rpm. No need to bother distro maintainers who are already doing so much thankless work.
Except this time, they seem to have made the decision because upstream was bothering them too much.
The thing about Linux filesystems, though, is that they consist of two parts: the kernel patch and the userspace tooling. Bcachefs is already in the kernel, so it's a bit awkward to leave out bcachefs-tools. Which is probably why it got packaged in the first place. Stable distros generally don't want loose ends flailing about in such a critical part of their system. If nobody wants to maintain bcachefs-tools for Debian, Debian will probably remove bcachefs entirely from their kernel as well.
I say this as someone who ran Gentoo for years and daily drives Arch today.
Because sometimes you don't want entire swaths of your server being rebuilt/tinkered with on a regular basis under the hood. "Move fast, break everything" is great in dev/test land, or a prod environment where the entire fleet is just containers treated like cattle, but contrary to what the SREs of the valley would believe, there's a whole ecosystem of 'stuff' out there that will never be containerized, where servers are still treated like pets, or rather, at least "cherished mules", that just do their job 24/7 and get the occasional required security updates/patches and then go right back to operating the same way they did last year.
This seems like a very basic concept for anyone purporting to do any sort of engineering or designing in any field.
If it were me, I would not be so eager to advertize my failure to understsnd such basics, let alone presume to call anyone else pretty dumb.
Note your examples are all bleeding edge / rolling distributions. Debian and the non-bleeding edge distributions go a different route and focus on reproduce-ability and security, among other things.
With the "get from crates.io" route, if someone compromises/hijacks a crate upstream, you're in trouble, immediately. By requiring vendoring of sources, that requires at least some level of manual actions by maintainers to get that compromised source in to the debian repositories to then be distributed out to the users. As you get in towards distributions like RHEL, they get even more cautious on this front.
On packaged operating systems (Debian, FreeBSD) - you have the system’s package manager to deal with (apt, pkg respectively). I can have an offline snapshot of _all_ packages that can be mirrors from one place.
IMHO, every programming language having its own package system is the weird thing.
The user experience is much better when working within tkese packaging systems.
You can control versions of software independent of the machine (or what distros ship).
Or in other words, the needs of software development and software distribution are different. You can squint and see similarities but the fill different roles.
Imagine that I make a simple OS-agnostic library in some programming language and want to publish it to allow others to use it. Do I need to package for every possible distro? That's a lot of work, and might still not cover everyone. And consider that I might not even use Linux!
A programming language will never get successful if that is what it takes to built up a community.
Moreover in the case of Rust distos are not even forced to build using crates.io. However the downside is that they have to package every single dependency version required, which due to the simplicity of publishing and updating them have become quite a lot and change much often than they would like.
The funny thing is that in the C/C++ world it's common to reimplement functionality due to the difficulty of using some dependencies for them. The result is not really different from vendoring dependencies, except for the reduced testing of those components, and this is completly acceptable to distros compared to vendoring. It makes no sense!
2. Because most Linux distros can only handle very primitive updates based on simple file overwrites, and keep calling everything impossible to secure if it can't be split and patched within limitations of C-oriented dynamic linker.
3. Because Linux distros have a very wide spread of library versions they support, and they often make arbitrary choices on which versions and which features are allowed, which is a burden for programmers who can't simply pick a library and use it, and need to deal with extra compatibility matrix of outdated buggy versions and disabled features.
From developer perspective with lang-specific packages
• Use 1 or 2 languages in the project, and only need to deal a couple of package repositories, which give the exact deps they want, and it works the same on every OS, including cross-compilation to mobile.
From developer perspective of using OS package managers:
• Different names of packages on each Linux distro, installed differently with different commands. There's no way to specify deps in a universal way. Each distro has a range of LTS/stable/testing flavors, each with a different version of library. Debian has super old useless versions that are so old it's worse than not having them, plus bugs reintroduced by removal of vendored patches.
• macOS users may not have any package manager, may have an obscure one, and even if they have the popular Homebrew, there's no guarantee they have the libs you want installed and kept up-to-date. pkg-config will give you temporary paths to precise library version, and unless you work around that, your binary will break when the lib is updated.
• Windows users are screwed. There are several fragmented package managers, which almost nobody has installed. They have few packages, and there's a lot of fiddly work required to make anything build and install properly.
• Supporting mobile platforms means cross-compilation, and you can't use your OS's package manager.
OS-level packaging suuuuuuuucks. When people say that dependency management in C and C++ is a nightmare, they mean the OS-level package managers are a nightmare.
A consequence of this is that they need to fix security issues in every library version that any application they distribute uses, including any statically-linked library. So, if they allow 30 applications to each have their own version of the same library, even Rust crates, and those versions all have a security issue, then the Debian team needs to find or patch themselves 30 different pieces of code. If instead they make sure that all 30 of those applications use the same version of a Rust crate, then they only need to patch one version. Maybe it's not 30 times less work, but it's definitely 10 times work. At the size of the Debian repos, this is an extremely significant difference.
Now, it could be that this commitment from Debian is foolish and should be done away with. I certainly don't think my OS vendor should be the one that maintains all the apps I use - I don't even understand the attraction of that. I do want the OS maintainer to handle the packaging of the base system components, and patch those as needed for every supported version and so on - and so I understand this requirement for them. And I would view bcachefs-tools as a base system component, so this requirement seems sane for it.
Sensing a pattern and it's not both Debian and the kernel. I mean can you even point at 2 more proven solid large and long term projects that have proven how to do it right?
"shared libs" was the product of storage constraints; a bobble in the smooth pace of progress that shouldn't be needed now. Our faith in "the way things are done now is the right way" and our (justified) fear of messing with the foundations of running systems will make excising the notion take longer.
Distros want to be able to mix & match upstream code with whatever version of the dependency they happen to have lying around in a drawer somewhere. Understandable, as the alternative is having to support dozens of versions of the same library.
Upstream developers want to use a single fixed version of their dependencies for each release. Understandable, as the alternative is having to support support a massive test matrix for every possible version of every dependency, just on the odd chance that some obscure distro wants to mix a brand-new libfoo with a 5-year-old libbar.
And all of this boils down to "versioning is actually really hard", because even with SemVer one's bug is another's feature[0] so there's still no unambiguous way to choose a version number, which in turn means upstream can't choose a version range for their dependencies and be reasonably sure that simply testing with the latest version won't result in breakage with other versions in that range.
If dependency management was easy, we would've solved it decades ago.
If you have a large number of statically linked program, you need to recompile all of them, which means that you need to know exactly which applications uses the affected library, when a bug shows up, and recompile all those applications and redeploy them.
Without knowing, I imagine that's also partly why Debian wants to have all programs use dependencies installed via APT, it makes it easier to keep track of which other packages needs to be rebuilt.
Personally I'm a huge fan of having applications ship with all their dependencies, like statically compile Go, Rust or Java jar/war/ear files or even containers. It's super easy to deploy, but you do need to be constantly rebuilding and keeping track of dependencies, and I'm not seeing a ton of people doing that.
For Python programs we use apt as a package manager for libraries, because that removed the burden of keeping on top of security issues, to some extend. We do sometimes need to back-port or build our own packages, but try to push those upstream whenever possible.
What the author left out in his blog post is that I specifically explained why this was going to be an issue and what was going to happen when he and I first talked about Debian packaging; and why for a package that needs to work correctly for the system to boot (fsck, mount helper, encryption unlock), playing these kinds of games with swapping dependencies out was not a good idea.
But with Debian and Fedora people alike it's been like talking to a brick wall. "No, your concerns don't matter, this is our policy and we're going to do what we want with your code". They've made it very clear that I have no voice or say in how they package it.
So then he breaks the build by switching to the packaged version of bindgen, an old unsupported version.
And he makes no attempt to debug, doesn't even send me the build error until months later.
As a result, Debian users were stuck on a broken version of -tools that wasn't passing mount options correctly, which meant that when a drive died and they needed to mount in degraded mode, they weren't able to. This happened to multiple people - that bothered to report it. This was quickly fixed upstream, but as far as I know it's still broken in Debian; and Jonathan Carter wasn't fielding any of those bug reports, I was.
I'm tired, frustrated and demoralized by this drama and tired of constantly having to respond to it. This has taken up an enormous amount of my time.
I really wish this guy hadn't packaged -tools for Debian in the first place. I never asked for it. I have enough on my plate to do without a bunch of distro drama to deal with as well.
> So then he breaks the build by switching to the packaged version of bindgen, an old unsupported version.
Could bcachefs-tools solve this problem upstream of distros by releasing a source tarball with the bindings already generated, so that you can use whatever version of bindgen you want, and bindgen is not needed at all when building from this release tarball?
This would be similar in nature to traditional autotools approach where some amount of pre-compilation is done to produce a release tarball from what actually lives in the source repository.
It's also what I do for some of my ffi crates. I have a script in the root of the repo which regenerates the bindings, but when published to crates.io, the bindings are already generated and bindgen is not part of the build downstream of my crate.
I'm not going to special case the bindgen bindings because this really isn't a bindgen specific issue - this could just as easily been any other dependency.
Any time a dependency is changed, the package version should be bumped and retested. This is a massive improvement in reproducibility, bisectability and our ability to QA.
Before this model, bisecting breakage that was a result of a change in library 'a' breaking something in consumer 'b' was effectively impossible; now it is.
What Debian and Fedora have been doing is a big step backwards, this whole 'unbundling of dependencies' (that were going to be statically linked anyways) needs to die.
They, and most others, have been doing that since their respectively beginnings, and for good reasons: everybody expects distros to fix security issues, which is greatly aided by ensuring an-as-small-as-necesary dep tree and thus a single version of any particular dep.
Changing that isn't impossible, but I am not aware of any distro that's managing this and be widely considered 'stable'.
How can distros change while keeping up their end which is stability and robustness in the face of bugs and security issues?
As I explained on another comment of mine (https://news.ycombinator.com/item?id=41409199), it's not since their beginning, but since a particular zlib security incident which showed them the risks of vendored libraries. Any discussion of vendoring policy which does not consider that incident is necessarily incomplete.
The distro people could've been working on the tooling for automating this at the distro level (and some people in debian are doing work that would enable this); we don't need to go this insane "unbundle everything" route.
IMO this is a huge mistake in the Linux distro world. It doesn't scale. It's similar to the weird idea that Debian should contain every program in the world and if the authors want their programs to be installable on Debian they must make a Debian package for it. And Fedora and Arch and Gentoo and...
Clearly bonkers.
You might consider adding a big warning on your official documentation about unsupported distribution packages.
Add links to relevant issue tracker/bug reports or mailing list discussions saying until So and so issue are resolved, the official statement is that distribution package is unsupported, not recommended and deemed dangerous to use.
This is as much leverage as you can have with the distribution community. Then wait for them to upstream patches and attempt to fix the issues. Accept fixes as you consider appropriate or not.
It's also important to at least respect the distribution's opinions in regards to having only one version of a library's major release . Just respect and understanding, you don't have to agree.
Also, all the problematic libs cited by the packager are 0.x.x numbered which sounds like a very young and immature ecosystem of dependencies. Of course, this is bound to cause pain to packagers. I think this speaks volumes about the high level of interest for bcachefs, that they actually tried to make it work instead of not bothering.
If debian is fine with packaging versions 2.0 and 4.0 but not 0.2 and 0.4, then debian does not understand rust version numbers.
What lessons have you drawn from it?
[1] https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag..., discussed at https://news.ycombinator.com/item?id=26203853 (320 points, 278 comments)
[2] https://drewdevault.com/2019/12/09/Developers-shouldnt-distr..., no notable HN discussions
[3] https://drewdevault.com/2021/09/27/Let-distros-do-their-job...., no notable HN discussions
they cannot see that by giving in to static libs you make every little application maintainer into a full-time distro maintainer responsible for it forever. ... good luck with that.
(The rest of the problem is that the goals of Debian are incompatible with upstream refusing to support versions of their own code more than a couple of months old. Which they transparently are, so there isn’t much to argue about here.)
https://github.com/koverstreet/bcachefs-tools/blob/b422b19f6...
Most C Debian packages have a few dependencies in my experience, they are just “standard”, because they are old and stable.
Then in principle, we could curate a set of useful Rust crates, treat the set as a single dependency, and be on the same page as C++ with Boost.
It doesn't require anything. You can write those things yourself if you want. And for some critical things you should.
But given the tiny number of languages with logging, uuids, and error reporting built in... what are you taking about? They're doing just fine.
This is a result of every Linux distros' self-centred requirement for the entire world to be packaged by them, and a refusal to accept vendoring.
Python has exactly the same problem. You can't even `pip install` on Debian any more because they insist on copying a random selection of Python packages into Apt and they can conflict.
And, they insist on not including venv by default, which IMO seems like a tiny little "fuck you" to users.
[0]:https://github.com/gentoo/gentoo/blob/master/sys-fs/bcachefs...
If the comparison is with C/C++ then the comparison is flawed:
- some you would never see in C/C++ because there's no point to them (e.g. safe bindings to system libraries)
- some are the same dependency but split in multiple crates to improve compile times (e.g. `anstyle`, `clap` and `regex`)
- some in C/C++ would just be vendored due to the difficulty of package management (see for example [0])
> This for some reason includes stuff like "winapi-i686-pc-windows-gnu" which really has no place for a set of linux software tools.
I don't know how Gentoo's build system works, but Cargo resolves the versions for *all dependencies*, including those that are conditionally compiled (e.g. based on the target OS). This is because the `Cargo.lock` file, which contains the actual dependencies versions used, is supposed to be versioned in e.g. git, and thus needs to be cross platform and be the same for everyone compiling the crate.
[0]:https://wiki.alopex.li/LetsBeRealAboutDependencies#gotta-go-...
Now, there is the separate concern of cargo-vendor downloading the winapi crate, which it does, also on Linux. It doesn't have to because winapi is never built, yet it still does it. That one I agree is a bit wasteful, but it's a known issue.
Those immature unproven little projects and their silly "best practices". What do Debian or the kernel know about building and maintaining bedrock software?
https://www.phoronix.com/news/Linus-Torvalds-Bcachefs-Regret...
Debian has a Rust policy, which among others prohibits this kind of extreme vendoring. Upstream insists on exact versioning (thus requiring vendoring), including hacked-up versions of some upstream crates, and also insists that Debian maintainers go on a war to change Debian's Rust policy to his liking (this is why I left #bcache; I don't set these policies, I'm not interested in fighting against them, and I'm not interested in an infinite rant about how I should spend my energy on such a fight). A couple of other people and myself made some attempts to get the Rust parts of bcachefs-tools buildable and working (and yes, it was really working, and no, the version changes led to no real breakage that I know of), but you can't reasonably package software from an upstream who is so hostile to everything about your distribution. If it's “my way or the highway” about everything, I guess it's the highway, then?
Maybe someone who cares deeply about bcachefs and cares deeply about Debian will pick it up, but that person would better have a really thick skin.