Asahi Linux lead developer Hector Martin resigns from Linux kernel

I think nvidia uses out-of-tree drivers that are dynamically loaded - does that mean that nvidia drivers are tied to specific kernel versions?

It seems like a huge technical factor holding back a stable ABI is the C compiler itself. Binaries changing between compiler versions and changing with different compiler flags.

So while your code can be written such that it appears to respect the interface of an external library, the underlying binary representation might not line up correctly.

If there is agreement that modularity is good for the kernel but technical limitations prevent that from being a reality - surely the solution is to improve C's interoperability first?

---

I'm not a kernel developer and am probably naive here, but on the surface it feels like offering a stable driver ABI is one possible solution to the rust-in-linux controversy that has lead to so many people exiting kernel development.

I'd imagine if projects like Asahi could simply offer out-of-tree drivers, they wouldn't need to maintain a kernel fork (at least not for drivers) or negotiate with core maintainers (which I understand is stressful).

Might also make it easier for vendors like Qualcomm/Samsung/Nvidia to distribute drivers out-of-tree, perhaps reducing the need for long running Linux forks and allowing devices to update to modern Linux kernels.

As a novice hacker, I'd imagine the ability to reuse proprietary driver blobs would allow distros to be created targeting hardware that was otherwise impossible to access as drivers were hidden behind custom kernel forks (e.g. install mainline Fedora on a Samsung phone, taking the GPU driver from the Samsung build of Android - or OpenWRT on an Android powered portable 5g modem).

Robigalia is also an effort, but one that hasn't been touched in nearly a decade.

Have there been any recent popular developments on a similar workflow that is as robust as e-mail ?

What? PRs or issues being forgotten happens all the time, especially for large projects.

This seems like a tu quoque fallacy. The feedback is either applicable or not, regardless of who said it. They're absolutely correct that Being toxic on the right side of an argument is still toxic.

Even if there is hypocrisy (whether judged by you personally or someone else), it wouldn't invalidate the point.

Certainly in context, this seems fairly reasonable: https://chaos.social/@sima/113961283260455876

Yeah this isn't about "being civil" or "friendly" and even less about "don't call out". This is about calling out in an ineffective and imprecise way, so that the people actually trying to change things are busy patching up your collateral damage instead of implementing real changes, while all you've achieved is airing your frustration for internet drama points.

When you're that harmful with your calling out, eventually I'm going to be fed up, and you get a live round shot across your bow.

And if that then causes you to ragequit, because you can't actually deal with the heat you've been dishing out coming back around the corner: fuck off.

Or as Dave Airlie put it upthread in this conversation: "Being toxic on the right side of an argument is still toxic, [...]"

So please do call out broken things, do change things, I've been doing it for well over a decade in the linux kernel. But do it in a way that you're not just becoming part of the problem and making it bigger.

---

And this is not the first time something like this has happened with Marcan. He may be tired of the Linux devs, but many of them are also tired with him (including some of the people working on Rust, it seems).

And this is part of a conversation on what went wrong here, not an attempt to rally the troops. You really can't compare it to Marcan's stuff. This kind of (selective) demand for absolute perfection is really not great.

Q: What would make you even more happy with Linux? GregKH: If you contribute to it.

Firefox and Chromium's bespoke tools have their pluses and minuses but they're a lot easier to deal with that the kernel "workflow".

Instead of providing helpful advice like outlining the current situation and suggesting specific improvements (action A, task B, and goal C) to reach the goal, it feels rude and offensive.

It was used to describe the Rust for Linux project, as well as any other potential efforts to bring other languages into the kernel, of which there are none. It is clear why someone working on the Rust for Linux project would feel that "this cancer" refers to the project that they are working on.

I'm not trying to pull out pitchforks, I don't want anyone to burn. I just want people to collaborate effectively and be happy, and I think it is empirically clear that calling something that grows/spreads and that you think is bad "cancer" is not useful, and only inflames things. It is not an illuminating metaphor.

It's almost like there are more important goals wrt software development.

Any change introduces anomalies and those cause all sorts of hell.

> And if you're wondering why you didn't realize this: It's impossible to change people by telling them they're wrong. What does tend to work is explaining different perspectives, so that they can figure it out themselves. And sometimes that's just way too subtle to ever register.

> "suck it up, buttercup" or "I hear you; rust is gone".

where various levels of compromise happens.

Maintainers are people and people can change their mind over time. If rust was a huge success in large parts of the kernel and you still had a few holdouts, sure, you could tell them to adapt or go away. In this early stage, it's kinda up to rust people to show that both they and rust can work in this setting

> As for how to move forward, (...) Either Linus takes the pull, and whatever Christoph says is irrelevant, or he doesn't, and R4L dies. Everything else is a waste of everyone's time and energy.

It does look like maintainers should have a "disagree and commit" mentality at some point, whatever decision they end up making.

I thought Rust in Linux was evaluated, discussed and agreed upon years ago. The fact that there are people still trying to sabotage it shows that they don't follow the "disagree and commit" principle.

They are more like "disagree and make the others lives a living hell until they bend to my will".

If you start with the assumption of a), there are no valid technical challenges to merging it. It's just better for everyone. Before Hellwig put his foot down as "not merging because rust sucks", he made a series of technical arguments against the patch, which were all transparently bullshit. It was those arguments that really raised such a furor, instead of all the other ways some C devs have disdained rust in the kernel in the past, because they were obviously made in bad faith. And when he was called out for them, he just went full "no rust in kernel".

"... I come at this from the perspective of having worked on Linux since around December of 1991. I first met you and Tove in 1995 at the Free Software Conference at MIT that Stallman sponsored. ...

Probably the last technical contribution of my career is leading an initiative to provide the Linux community a generic security modeling architecture. Not to supplant or replace anything currently being done, but to provide a flexible alternative to the development of alternate and/or customized workload models, particularly in this era of machine learning and modeling.

Four patch series over two years, as of yesterday, not a single line of code ever reviewed. ...

We were meticulous in our submissions to avoid wasting maintainers time. We even waited two months without hearing a word before we sent an inquiry as to the status of one of the submissions. We were told, rather curtly, that anything we sent would likely be ignored if we ever inquired about them.

We tried to engage, perhaps to excess, in technical discussions attempting to explain why and how we chose to implement what we were proposing. ...

There were never any relevant technical exchanges. The discussion consisted of, we have decided to do things a certain way, no discussion, if you don't like that you should really consider doing something other than submitting to upstream Linux."

Yikes, this is EXACTLY what Linus was talking about except an order of magnitude worse!

Speaking as an actual People Manager, his ideas sound pretty shallow.

The current management model is based on consensus in some parts, and a dictatorship in others. Its not clear which parts are which because of sprawl, and non-responsiveness.

This is a core communication people issue. The problems arise because in consensus models, people are expected to adopt archetypical roles based somewhat on preference, but mostly on the needs of the group, towards productivity.

If no one else is doing that type of role, someone will be forced into doing it, and will adopt characteristics of that role, regardless of their personal opinions about it.

When they get punished for fulfilling the role, you lose people.

> doesn't mean we should throw our hands up and consider things "unfixable" either.

The given structure is unfixable, because it sets the stage for a trauma loop where actions can't take place without people who are expected to fulfill a role to meet consensus, but in doing so they get punished for volunteering.

There is no way around this structure so long as you use a consensus model in whole or part, in dysfunctional groups the people involved either fail to communicate or worse there are members that impose coercive cost by assuming the unproductive roles so no action happens while resources are wasted.

This really is basic stuff covered in your college level intro to communications coursework.

For those wanting to know more about this, you can read more about the roles at the link below.

https://pressbooks.pub/smallgroup/chapter/roles/

    > there is the tangible risk of stifling innovation and Linux is the only place that innovation can occur in the operating system space.

That might reflect my environment, but all of the things I’d consider C variations for I’d use Rust for as well…

I'm amazed that Hector is doing this fully in public using his real name.

Linus should have stepped in long before a maintainer blew their stack and started throwing out ultimatums. Once that happened, Linus could have still stopped everything with one sentence -- "Let me look into this.", but he did not.

Linus only got an opinion once things blew up on social media which proves that social media works which is the exact opposite of what he says he wants (and will just encourage more of the same).

That shouldn't be too surprising - I mean, its an old project with a whole lot of technical baggage. Projects tend to slow down over time. And that is legitimately really frustrating when you want to shake things up or push for change. I would be rubbish as a linux kernel developer. I have the wrong temperament for it.

There's a reason why some tech companies interview for both technical skill and culture fit. Sounds like he's got the technical chops, but he's not a good fit for linux.

And when you're in a situation like that, your choices are essentially Voice or Exit. Voice is to do what he's tried to do - kick up a fuss about the problems to try and get them fixed. Thats a skill on its own - and it sounds like he's not been super effective at that. The other option is Exit. Which of course - sensibly, he's now done.

> he has gotta Stop Posting and keep those kinds of thoughts away from his keyboard.

Nah. Bottling this stuff up is a bad long term play. You end up getting bitter, cynical and resentful. I think we've all worked with people like that, and its miserable - both for the person and for their coworkers. I think its better to shoot your shot. Even if you miss - as he has here - you learn a lot about yourself and the world. And there's no shortage of interesting projects out there to work on. Pick something that matches your speed.

You don't even take time to figure out who's the commit author.

From shaming everything else as either slow or unsafe to rewrite the universe, the Rust community makes it hard for new comers to consider the language by its merit.

But the good thing is, the hype has settled down and Rust has found its niche. It’s not tackling Go or Python anytime soon and competes in a different plane.

Zig is another great alternative for someone like me who never found Rust a pleasant language to work with.

First of all because we're talking about the kernel community here, which was incredibly toxic and dramatic long before Rust even existed. Linus has chilled out in the past few years but that legacy isn't entirely gone.

C++ drama has nearly come to fistfights at conferences, and the only reason that doesn't get talked about more is that the toxicity stays mostly on private (not public) mailing lists as a result of the more insular nature of that ecosystem. Nowadays you have a lot of people just quitting over things like the fact that a member of the standard committee was convicted on CSAM charges.

There are plenty of "C supremacists" in the tech influencer community and the maintainer mailing list of every distribution.

And streamers like PrimeTime that eagerly jump on every opportunity to shit on Rust and actively dump gasoline on every flareup of drama for ad revenue.

In general, programmers seem to love being elitist about languages and tools. Remember how everyone used to dump on PHP constantly?

There seems to be an entire second world of “Rust community” and Rust zealots online who are heavy on the drama, though. It really does feel like an alternate Rust universe.

Although when I think about it, several of my other interests and hobbies are exactly like this. Great in the real world, but as soon as you step into certain online spaces it’s drama and toxicity.

In this specific case, I think this is more about kernel drama than Rust drama.

If rust ends up being mainstream successful, those people will move on to something else and start attacking rust for whatever they feel their new language is superior in

Someone should have hugged them when they were kids

> The common ground is that I have absolutely no interest in helping to spread a multi-language code base. I absolutely support using Rust in new codebase, but I do not at all in Linux.

[0] https://lwn.net/ml/all/20250204052916.GA28741@lst.de/

>Every additional bit that another language creeps in drastically reduces the maintainability of the kernel as an integrated project. The only reason Linux managed to survive so long is by not having internal boundaries, and adding another language completely breaks this. You might not like my answer, but I will do everything I can do to stop this.

I think that clearly means he is out to sabotage any other language than C making it into Linux.

The rust for linux project exists because Linus approved it. It's that simple. Linus thinks it a good idea to at least try Rust in the kernel. If he didn't, then none of this would be happening.

I was just pointing out that the positions are much more binary and un-nuanced than the previous poster was claiming, even in the thread in question. I'd bet the DMA maintainer is not the only one who holds this dogmatic position, either.

I'll also note that the complaints from this maintainer aren't even social though. He is very explicit in his reasoning: two+ languages bad, single language good. There's clearly little that will change this, other than working around him (though, again, I agree that social media blackmail certainly won't improve anything).

Maintainers do steer direction of development though. A lot comes from maintainer saying "we are not accepting XYZ".

Today we only have proper open source GPU drivers because people like David Airlie who stand for their principle against likes of Nvidia and AMD.

Blatant NIMBYism is the problem here and you cannot reduce it by accepting everything.

I don't know what this is about. In my experience, refactorings that change the semantics of APIs are much easier in Rust than in C. E.g., change assumptions about the lifetimes of pointers passed into APIs: the Rust compiler will tell you where you need to change anything; the C compiler will happily compile your code and you'll corrupt memory at runtime.

And this is the reason why it's always regressing and never moves beyond alpha.

One language per repo rules makes sense when you can have many smaller repos, but for something as immense as the Linux monorepo it's really limiting. Especially considering the lack of desire to add stable interfaces from which other repos could operate independently.

> Which is a valid viewpoint. Let's not pretend that's not a technical argument.

So if Linus shouldn't overrule his deputies, and one deputy can completely block something in a subsystem that isn't theirs and has explicitly stated to do "everything in their power" to stop it, what exactly can the community "figure out" to get it to proceed? His decision not to get involved literally makes it impossible for the situation to change, so the confusion is why it's being phrased as if it's anything other than that.

If Linus were to come out and say "I changed my mind, I no longer think it's worth pursing trying to integrate Rust in the kernel due given the issues at hand" or even "regardless of my personal views, I don't have any desire to change the current system we have for how code gets merged into subsystems or who has the ability to block it, so I'm not going to overrule this", it would make a lot more sense to me, but by making it sound like there's anything left to figure out when someone with veto power has a stated intent to stop things from moving forward is just going to let the issue fester and produce more frustration on both sides. It almost seems inevitable that any future discussions will spiral out of control; there's no other way for it to conclude than either Linus overruling his deputy or the experiment just ending as a failure at this point, so he might as well just make that decision now rather than later.

The type-system analysis of Rust is smart, but not restricted to the language per se, see https://github.com/ityonemo/clr. One merely has to have proper namespacing and necessary type info from generic code to do annotations and solve them. These things are solved in Rust via trait system.

Retroactively patching C to have namespaces will not work and same holds for generics, meaning concrete how to attach lifetimes to generic code.

> Zig went in a different direction.

There is stuff cooking for debugging comptime and better than lsp infos, but this is only wip and hearsay. Might be enough to write external static analysis or not.

Correct me, if wrong etc.

We don't even have a C++ killer language yet. These things move very slowly.

Both OSes are huge C++ codebases. Features being disabled or forbidden is a non-userland thing (basically only because resources are constrained. 20KB of exception runtime bload isn't the same thing when you only have 200KB of available RAM vs. when you have 20MB+)

My opinion (and the opinion of people using C++ in such constrained envs.):

* C++ (the language itself along with stuff like <type_traits> and other freestanding headers) is the best thing since sliced bread. So many ways of reducing boilerplate while both writing safe code (RAII) and being mostly in control of produced asm

* most of the other parts of the stdlib are defective

* the committee process sucks, but fortunately compiler vendors allow us to dodge their bad decisions (making #embed available for C++, -fno-exceptions, etc.)

(Edit: Ok, reading some other context, it seems like this is indeed what's happening, which does make this reaction a little less justified. I can certainly see why it's no fun to have people in the project saying they will do whatever they can to stop an initiative, but if they aren't actually able to block it, then it doesn't seem worth quitting over)

I used to know C. It was my first language, and I spent a fair bit of time in it, but that was before compilers started to treat undefined behaviour as an excuse to break reality.

Arguably I'm not smart enough, or disciplined enough, to write C. This may be the case, and therefore I have naturally focused on other languages. Rust, to no-one's surprise, is my current favorite. I've never contributed to the kernel — for more reasons than just the language — but the language would be reason enough on its own.

C programmers are getting rare...

And that particular side was told by Linus Torvalds himself that his way of Social Media brigading is inappropriate for kernel development, and that the current process works.

I choose to believe Linus. His work on and guidance of the kernel used to and still does work pretty great.

Suppose you want Rust, just not a single Rust driver using PCI. But CONFIG_PCI and CONFIG_RUST both selected will cause the PCI abstractions to get built anyway, even if not a single driver using them is selected. Then if the PCI subsystem introduces a change but the Rust counterpart fails to follow it fast enough, the build will break for the kernel as a whole.

This was illustrated with an example in that email thread.

I have some sad experience with polyglot projects. Unless you enjoy pain and drudgery, it’s an extremely unrewarding thing requiring very careful treading.

This leads to two situations: the C maintainer has to also update the Rust binding, or the patch has to wait for a Rust capable maintainer to fix them up.

For the first solution, the C maintainer is claiming he has enough work on his plate just keeping up with the C code that he doesn't want to work on Rust. He also believes it is unfair to force every C maintainer to learn enough Rust to keep their integration points working.

For the second solution, the C maintainer is pointing out that Linus has a history of refusing to merge C patches that break Rust builds. So even though the Rust advocates promise they will handle all of the Rust stuff and they are OK with their stuff being broken after C changes until they get around to fixing it, Linus doesn't seem to actually allow for this in practice.

I can see the maintainers point. Sooner or later Rust bindings on critical systems will be essential and changes to the C code will be gated on changes to the Rust code. It is a fantasy of the Rust maintainers that they will have to shoulder that entire burden. But even if they did, it would give them some pretty hefty leverage over the C maintainers in those essential cases. And it also introduces a requirement that someone with enough Rust and subsystem knowledge is even available for the job. People making promises today may disappear tomorrow, leaving the maintainer with an essential binding to a language he doesn't want to support.

edit: as has been said elsewhere, Linus could make some of this go away by committing to let the Rust builds break. I don't know if that could be done in a way that wouldn't be a de facto fork.

Much like the “don’t rewrite from scratch” mantra was written in blood and tears.

They reiterated that in the thread too.