We are the "thin blue line" that is trying to keep the code high quality
lore.kernel.org
lore.kernel.org
Resigning as Asahi Linux project lead - https://news.ycombinator.com/item?id=43036904 - Feb 2025 (826 comments)
Asahi Linux lead developer Hector Martin resigns from Linux kernel - https://news.ycombinator.com/item?id=42972062 - Feb 2025 (1015 comments)
Me stepping down as a nouveau kernel maintainer – https://lists.freedesktop.org/archives/nouveau/2025-February...
Despite incredible effort from maintainers, getting necessary changes into Linux can take forever. In the subsystem I depend on (and occasionally contribute to directly) it’s kinda assumed it will take at least a year (probably two) for any substantial project to get merged. This continuously disappoints PMs and Leadership. A lot of people, understandably, chafe against this lack of agility.
OTOH, I’ve been on the other side of kernel bugs. Most recently, a memory arithmetic bug was causing corruption, and took my team at least an engineer year to track down. This makes me quite sympathetic to maintainers demands for quality.
I’ve also been on the other side of the calibration discussions where Open Source work goes under appreciated. The irony never stops (“They won’t merge our patches!” “Are you having your engineers review theirs?”). That and the raw pipeline issues for maintainers (it takes a lot of experience to be a maintainer, which implies spending a lot of a bright engineer’s time on reviewing and contributing upstream to things unrelated to immediate priorities).
I would love to see Linux thoroughly and meaningfully tested. For some parts it's just... hard. (If anyone wants to get their start writing kernel code, have a crack at writing some self-tests for a component that looks complicated. The relevant maintainer will probably be excited to see literally anyone writing tests.)
For this particular bug, the cheapest spot to catch the issue would have been code review. In a normal code base, the next cheapest would have been unit testing, though, in this situation, that may not have caught it given that the underlying bug required someone to break the contract of a function (one part of Linux broke the contract of another. Why did it not BUG_ON for that...).
Eliminating the class of issue required fairly invasive forms of introspection on VMs running a custom module. Sure, we did that... eventually.
Finding it originally required stumbling on a distro of Linux that accidentally manifested the corruption visibly (about once per 50ish 30 minute integration test runs, which is pretty frequently in the scheme of corruption bugs).
Technically, the borrow checker and bounds checks wouldn’t have done it here (I’m aware I’m being obtuse by not just linking the bug).
Having cleaner types and abstractions would almost certainly have solved the problem though. Normal C++ would have worked as well as Rust.
Egcs vs. gcc was a big deal back in the day and in the end we ended up with gcc by the egcs guys and that was it. When you win, everyone forgets the 'drama' existed. When you lose, everyone remembers you as just the drama guy. RMS had the drama label for decades. Things are not even different. They're the same again. It's like when you'd buy those Chinese NES dupes and they'd have 999 levels of Mario but half the levels would be the same but with different colour bricks. Isomorphic to original but distinct. That's this story.
The new guys definitely want to make things different, but it seems there is a lot of debate over whether it will actually be better. Really, they should just write their own their kernel. If rust is really that much better, they'll will.
Edit: This seems like more of an indication of a culture that lacks effective conflict resolution than any kind of technical question.
You are missing the whole point here. The kernel is a survival epic amongst millions of other failed projects. You don't get to tell the old captain and its lieutenants how to nail the planks and helm the ship when you just went pass the Titanic and Britannic wrecks because metal is so cool.
They're "old-school" because they have to be. Engineers will excrete their pet project and then leave and now they will have to support it. They are mean because that's the only "power" they have, as is explained in the post.
I'll leave you with this quote from "The Night Watch" by James Mickens (https://www.usenix.org/system/files/1311_05-08_mickens.pdf)
> This is not the world of the systems hacker. When you debug a distributed system or an OS kernel, you do it Texas-style. You gather some mean, stoic people, people who have seen things die, and you get some primitive tools, like a compass and a rucksack and a stick that’s pointed on one end, and you walk into the wilderness and you look for trouble, possibly while using chewing tobacco. As a systems hacker, you must be pre- pared to do savage things, unspeakable things, to kill runaway threads with your bare hands, to write directly to network ports using telnet and an old copy of an RFC that you found in the Vatican. When you debug systems code, there are no high- level debates about font choices and the best kind of turquoise, because this is the Old Testament, an angry and monochro- matic world, and it doesn’t matter whether your Arial is Bold or Condensed when people are covered in boils and pestilence and Egyptian pharaoh oppression. HCI people discover bugs by receiving a concerned email from their therapist. Systems people discover bugs by waking up and discovering that their first-born children are missing and “ETIMEDOUT ” has been written in blood on the wall. What is despair? I have known it—hear my song. Despair is when you’re debugging a kernel driver and you look at a mem- ory dump and you see that a pointer has a value of 7. THERE IS NO HARDWARE ARCHITECTURE THAT IS ALIGNED ON 7. Furthermore, 7 IS TOO SMALL AND ONLY EVIL CODE WOULD TRY TO ACCESS SMALL NUMBER MEMORY. Misaligned, small-number memory accesses have stolen decades from my life. The only things worse than misaligned, small-number memory accesses are accesses with aligned buf- fer pointers, but impossibly large buffer lengths. Nothing ruins a Friday at 5 P.M. faster than taking one last pass through the log file and discovering a word-aligned buffer address, but a buffer length of NUMBER OF ELECTRONS IN THE UNI- VERSE.
I have to work with a "kernel a-hole". Sure it sucks but when there's some mutex bug in a chip driver, he's the one we can call and fix the problem and push the fix upstream even though he's already overworked. He's an a-hole but I respect his rules because he's the one suffering and pulling in the work and I don't.
Yes the kernel is stagnating (patches and email mailing list, line length limited to be compatible with old tty terminal, etc.) but when you're at the helm of a ship you don't just steer it near the nice looking island just because it looks pretty and there doesn't seem to have rocks.
Yes, it works, but there's a very real opportunity cost paid every day, month, release. It works until it doesn't. (Marcan crashed and burned out. But sure it works. And ... after all, since there's no real alternative we just pretend that it works. So as I was saying, it works! It's alive!)
Of course it's not easy to set boundaries, but the whole "patches are welcome" sets up false hope and masks the real incompetence in managing the kernel. (And of course is the focal point of the broader, very human tragedy of the Linux and FOSS ecosystem. insert obiligatory xkcd 2347 link here)
Only 13.3% of all changesets are from people that are either unemployed or their job is unknown [1].
Even greg k-h said that most of the development is backed by big companies so the argument of 'poor unpaid maintainers' doesn't go too far.
didn't the good egcs stuff get merged after all?
EGCS took over as mainline, and what good stuff there was in the old mainline got merged into it. But it took sustaining the fork for years to make that happen.
The new school guys greatly underappreciate the wisdom of why and instead try to change things without first understanding.
The fact that the patch wraps kernel/DMA is why he was CC'd in the first place, but that doesn't give him authority to unilaterally reject it any more than he would have the authority to reject an Nvidia driver for using DMA.
What would happen in that case (and is likely to also happen in this one) is that CH's objection will be completely ignored, because it's absurd. Maintaining the DMA subsystem doesn't give you veto rights against every driver or subsystem that needs to use it.
The old school guys do things for a certain reason that young contributors often don't appreciate. Enough violations of that, and you're often ignored. Perhaps thats what has happened here?
Clearly this is a communication problem more than anything else.
https://lore.kernel.org/rust-for-linux/2b9b75d1-eb8e-494a-b0...
I'd be pretty upset if I was working on Asahi since the Linux project has basically bait and switched them after an enormous amount of work has been invested.
Now it is certainly somewhat the fault of the maintainers themselves for turning off thousand if not tens of thousands of eager, well-intentioned wannabe contributors over the decades, if not through their attitudes and lack of interpersonal skills, then through impenetrable build systems and hostility towards ergonomic changes.
But forget the eager amateurs - it is unconscionable that major technology companies & cloud providers don't each have damn near an army helping out with Linux and similar technologies - even the parts that do not directly benefit them! - instead of just shoving it into servers so they can target ads for cheap plastic crap 0.000000001% better than they did last week.
Greg pointed out in that email thread that:
> over 80% of the contributions come from company-funded developers. [1]
[1] https://lore.kernel.org/lkml/2025020738-observant-rocklike-7...
I posted that statistic to demonstrate that the kernal project isn't a typical underfunded, almost purely volunteer open source project as implied by many of the kneejerk comments here.
... they have, and they are selling that as premium. it's the classic "open core" model for the cloud era.
Ok. But if you make it too difficult for new developers to contribute, and the experience too stressful, then no one new will stick around to be long time maintainers, and when the old ones retire or die, there won't be anyone to replace them.
This was painful for me to read as someone who has seen how corporations think about “Open Source” - he isn’t wrong at all
Some maintainers are rejecting changes as a way to block a project they disagree with - ie. there is no path forward for the contributor. I wouldn't assume this normally, but they're not hiding this fact, it's self proclaimed.
On the other hand, Linus has been largely in favour of the R4L project, and gave the green light for it to go ahead.
The Linux kernel maintainers need to figure out among themselves whether they want to allow Rust to be introduced or not, and if so under what constraints. If they can't come to an agreement, they're wasting everyone's time.
Once that happens, either the project is canned, or people can stop arguing over if these changes should be upstreamed, and start arguing over how instead, and that's a lot more productive.
It's yours.
If you need someone to do free work but you can't convince them then you either get their boss to tell them to do it, you take over their job, or - the one thing that no one under 30 ever seems to do - fork and do the work yourself without anyone stopping you.
It is the maintainers job to determine whether a project should be pursued at all.
If a maintainer says "yes I'd be happy to accept feature X", and then simply rejects all PRs implementing X without giving feedback then they're a bad maintainer!
That's essentially what's happening here due to the fact that the kernel maintainers have not come to a coherent decision regarding R4L.
If you're asking them to accept your code then yes, you' are asking them to support your project forever.
If you weren't sending them patches then they couldn't block you. Since you are they can.
Again, it's not their job to support this great idea you have that means they have to change how they've done everything for the last 30 years.
That is to say, if you are building an tool as a collaborative open source project, then that implies that you intend to collaborate.
It works both ways. If you have a beef, don't be surprised if someone responds to it in a way you don't expect or like.
So far, I haven't seen discussions from both sides of the wall. But I hear a lot of noise from one side of the wall trying to get the attention of people on the other side of the wall. Now we will see if the people inside the wall think there is an issue with the gate.
Then say that. A big part of the issue is that there has been mixed signals from the Linux maintainers about R4L. Linus seemed to have supported it, and others are trying to block it.
The maintainers should get on the same page about the inclusion of rustlang code, and communicate that.
> If you weren't sending them patches then they couldn't block you. Since you are they can.
One of the big turning points of this drama was a maintainer from a different area (who had been CCed on the threads, but was not the person the patch was being submitted to) blocking a patch that they weren't going to have to maintain.
But saying that he is't going to have have any additional maintenance burden just because the rust code using his subsystem is in a different subtree is also not an honest assesment of the situation. That's not how the kernel developent works.
In the related submission on this topic [1], the author makes this argument in a lot more detail, that it's essentially impossible to make a Linux fork sustainable without massive investment that no one can realistically obtain.
I do also understand the frustration when an open source project strings you along (me: can I join your club?), ask for this and that (them: sure if you pay your dues), ensuring your cover every "important" person's pet use case (them: and also buy us lunch), then publicly snark about the whole thing (them: this new guy know about no mustard on the lunch!) and kill your failing motivation (me: oh, sorry).
That's why the fork is so appealing. If it's good enough for long enough you get to join the club.
If Rust truly is zero overhead on the C side of things then they should be just able to fork Linux indefinitely without any worry about what upstream does. Just fast forward every change and you're golden.
Then they can write every driver they can imagine to their hearts' content.
The fact they aren't doing this tells me that the promise that this won't impact C people at all isn't much of one.
It's like telling people who want JPEGXL in Firefox to "just" fork the browser, ignoring the massive extra effort that you actually have to convince everybody to use your fork instead of the original.
Talk about aggressively missing the point.
Rust developers are promising that rust in the kernel should have no impact on c in the kernel.
If rust in the kernel has no impact in c then you should be able to remove the rust from the kernel and move it to it's own project, since it will have no impact on the c either way.
This is primarily around Rust filesystem drivers. If you want to run a filesystem but it's implemented in Rust, you'd have to use that fork. If another person had the same issue (say a GPU driver that some maintainer decided they didn't like because of the country of origin or some other petty reason), then that would have to be a fork as well. Suddenly, you can't use that GPU with a Rust-implemented filesystem, because you have to pick one of the forks, or you have to make your own merged kernel from the two.
"Just fork it" doesn't work here. It's a logistical nightmare.
From the other side, as a very occasional contributor, I'd actually want to deal with fixes or reviews around the code I contribute.
But it's usually edge cases on otherwise stable and mature libraries, so hopefully it probably won't happen more than say once in a decade. If I got a mention on a PR I'd reappear, but that doesn't sound like the standard way, I never got involved again on anything submitted.
I feel like either the maintainers are doing an incredibly good job at vetting the PRs, or got used to deal with the aftermath another way and just don't need the original code submitter to reappear most of the time ?
Am I missing some bigger part of it ?
I empathize but we need to have people take over these initiatives and refactor them into something easier the maintain. Not saying introduce a new language or anything but change how we fundamentally look at “The Kernel”. I think reducing scope and making it so hardware providers must maintain their drivers is a good start. If your toolchain doesn’t suffice, create a new tool.
If rust is so much better, create a new kernel in rust and force a paradigm shift. I think we can do better than bicker and fight over it and post email chains about it. Bring solutions. Tech debt is just OpEx to everyone else.
Rust-only ecosystem would be pretty cool though. It may be worth just forking linux for the sake of compatibility, and keeping the license going. I don't see a future otherwise.
I'd also like to see better compiler diversity. Maybe once gccrs rolls around we will see different attitudes around rust emerge, compared to C/C++ which have more distributed development.
I figure it's because the companies and orgs that started working with and investing in the rust ecosystem start by contributing to the compiler, which is non-copyleft and just try to extend that because it works to their advantage. You see this with a lot of languages/ecosystems with corporate sponsors, but also just because it gives you a unique selling point in an area where copyleft software dominates. Like, there will always be some company (like imagine a defense company) that absolutely refuses to publish all of their changes, so that represents a niche that can sustain smaller permissive projects.
I think that for an individual corporation, permissive is better and they won't make a decision to go copyleft unless forced. But for the ecosystem as a whole, GPL is better for business (especially when it comes to something like a kernel) because it forces companies to publicly fork over their drivers and collaborate, and reduces the competitive advantage vs companies that would otherwise not publish their changes if it was permissive.
That was probably too many words. Anyways, it would be nice to see some of this improved, i think it's a great language and will probably replace C++.
Both copyleft and copyright (to a lesser extent) are in opposition to my beliefs and goals.
Most of them at some point have been stung by finding a library that perfectly solves their problem, only to notice that it’s licensed GPLv3 (or similar) and thus the legal dept won’t let them touch it with a ten foot pole (and/or interfacing with that dept is not worth the bother). They’d rather not put others in that predicament and want their work to be useful to as wide of a spectrum of devs as possible.
How do you propose that happens?
What if there was a rust compiler to C, that produced like readable, compliant C code for the kernel. And then developers that want to work in rust can publish their original rust code to some third party location, so the reviewers have no idea if the C code they receive it was originally written in rust or not.
Try what you said with even well align languages like TS and JS and it would be hard to work with.
And then what?
A kernel by itself isn't very useful. Even if your kernel is somehow superior in every way, you need software to target it, so to be a successful replacement for linux you basically have to be completely compatible with any software that currently runs on linux, which not only means a massive amount of work just to keep up with changes in linux, but constraints on your own design.
And then there is hardware support. If you are a fledgling project, how do you get hardware vendors to write drivers for you?
Yet he can't even put up his real name. Pathetic
FOSS evolves past its chokepoints by forking.
So, a credible group of Rustaceans and their backers need to come up with a plan to do it.
It doesn't have to be antagonistic -- it's exploratory. If it works out well, it's a lot easier to adopt once the imagined issues are resolved or evaporated.
Subsystem by subsystem would be my suggestion. And just do it. And sooner or later, it will be (a) good enough that it's pulled into -next, or (b) they'll give up because it's not worth the effort.
I'd be pretty surprised if, for instance, one of Google/Amazon/Meta/Microsoft/Cloudflare/Netflix/etc wasn't interested in an ABI-compatible kernel written in Rust. Get a few biggish backers, and LF would possibly even adopt it.
Unless you've been already invited into the project?
Just because some people have invited you in doesn’t mean everyone is welcoming.
Where exactly? Yes, Christoph Hellwig did lay down in the street and did say "Over my dead body!", but everyone else just got on with it.
Do you really think a few nasty messages from a few nasty kernel maintainers is what stops Rust in the Linux kernel?
> Just because some people have invited you in doesn’t mean everyone is welcoming.
Exactly, but as much as open source can be a social thing, the Linux kernel is big business. And I think the people who really pay for the kernel really want Rust in the kernel.
https://www.youtube.com/watch?v=WiPp9YEBV0Q&t=1529s
There was a reasonable discussion about Rust, and then T'so drops the bombshell:
"I suspect part of the problem here is you're trying to convince everyone to switch over to the religion as promulgated by Rust and the reality is that ain't going to happen because we have 50 plus file systems in Linux they will not all be instantaneously converted over to rust" [26:05]
He pretty much shouts at the Rust developers, denigrates their efforts and when they explain what they are trying to do which is to understand the semantic changes.
And then T'so then starts yelling at him even more. He wonders why people are not happy with his attitude - then perhaps he should consider how he communicates. The Rust guy is trying to be constructive and T'so was not.
> including an upstream language community which refuses to make any kind of backwards compatibility guarantees, and which is actively hostile to a second Rust compiler implementation (I suspect because it might limit their ability to make arbitrary backwards-incompatble language changes).
with the full context of all of this it is quite obvious this is just a classic case of a software engineer being an antisocial jerk with strange vendettas in their head and who thinks the strange personal opinions they came up with in their basement are unassailable facts
The guy is a poster who just can't help himself! Both points are demonstrably false.
>The thin blue line U.S. flag has been banned by some police departments in the United States for its associations with ideologies described as "undemocratic, racist, and bigoted."
>According to a 2018 law review article, "thin blue line" also refers to an unwritten code of silence used to cover up police misconduct, also known as the blue wall of silence, a term dating back to 1978
We still let people use the Swastika if they had a harmless tradition of using it beforetime. But if anybody new uses it, we assume it's because of the murder.
I don't know how Tso was using it and don't care nor have any reason to defend him, don't mistake me.
You mean its popularisation and heavy use by William Henry Parker III, Chief of the Los Angeles Police Department (from 1950 to 1966)?
He very much used it in a loaded manner to portray police as bastions of good and the last defence against "the criminal element".
From his wikipedia page:
Parker himself was known for his "unambiguous racism".
Do you think that Rowan Atkinson was dogwhistling about undemocratic, racist, bigoted police violence when he named his police sitcom "The Thin Blue Line" or do you think perhaps your dislike of the police has led to you associating negative attributes with anyone that happens to use any phrase related to the police?
Where I am from, the police is a respected, largely unarmed institution. Imperfect, as all institutions are. American left-political dislike of law enforcement isn't universal and most people don't have instant negative mental associations with everything police-related...
No, but Ben Elton certainly was when he created and wrote it, even before hiring Atkinson to star in it . . .
When I last spoke to him Elton didn't view the entire UK Police force with disdain but he absolutely felt it was riddled with clusters of bigoted and violent police. You can see that in his other works such as the The Young Ones and various novels.
are you? what for?
> but have you actually seen the program?
Yes.
> There isn't a hint of any of thise feelings.
It comes some 15 years after his angriest socialist days .. and it's a well crafted comedy played for laughs. You'll recall, I trust, that it mocks the police pretty heavily.
Detective Inspector Derek Grim isn't a lovable pussycat.
Have you seen it? Did you read the credits? What made you think Rowan created the show?
>far-right authoritarianism
Get a grip. Even if it were a reference to what you say it is, that has nothing to do with the "far right". It is stock standard centre-right (and centre-left for that matter) position in the US to support the police in principle (if not, obviously, in every action theyve ever taken). The "defund the police" types are a minority of a minority.
[0] how do you think police get so biased against everyone else to begin with? they're effectively dealing with the shittiest rungs of society on repeat, so they form a pattern and end up applying it to everyone they meet
So yes, to make it concrete and cut past the bullshit euphemisms, sometimes the police will shoot an "unarmed" black man. Sometimes, very rarely, the so-called "unarmed" person not only doesn't have a gun but doesn't pose a real threat to the life or limb of another person. That sucks. But we cannot get rid of those "type I errors" without curtailing the use of force in a way that produces more "type II errors" in the sense that the police don't use force in a scenario where they should have, and someone gets hurt unnecessarily by someone that should have been disarmed or killed by a policeman earlier.
That is in no sense a defence of those extremely rare cases where a policeman just murders someone and there is no justification or excuse.
It is not unreasonable for them to point out that if they are held to an impossible standard where they get blamed if they don't perfectly protect the public but they also get blamed if they ever have to make a split second decision with limited information, choose to use force, and it turns out not to have been necessary, that that impossible standard is not fair.
None of that has anything to do with "neofascism" or "authoritarianism", which are simply bogeymen invented by the left.
Whom do they serve?
Within capitalist societies police protect the interests of the state, which itself exists to serve and benefit the capitalist class. Police "maintain order" to ensure the proletariat continues to submit to the state and participate in the capitalist machine, and they commit violence against anyone who does otherwise.
In fact, don't even sleep inside your own home!
Maintainers seem more like HOA boards than police.
you want your merges in? patch it in your own branch
the real crux of the issue is the quality of "the offical version", the real issue, then, is about official branding.
I remember back in the day there were multiple kernel versions by different mantainers.... then again I'm probably missing the forest for the trees or something
> Out of tree doesn't work. People, both in the kernel and out of the kernel, want things in tree.
https://lore.kernel.org/lkml/1e8452ab-613a-4c85-adc0-0c4a293...
The Linux kernel quite intentionally makes maintaining a fork a pain in the ass to pressure vendors to mainline their drives. But now mainline maintainers are refusing to accept those drivers. So it's unlikely we will ever see Macbooks properly supported on Linux.
This is why forks made for specific devices are seldom updated for more than a few months (or a couple of years if you're very lucky).
I feel like this problem should have a technical solution.
maybe a VCS good enough that this isn't as big an issue? (along the lines of pijul?) or
maybe something from formal verification methods to better enforce or more clearly explain why some drivers cannot be accepted. then again, I don't think the computer science is quite there yet
If your code is out-of-tree, every time someone changes an API that break your build, it's your responsibility to update your for and fix it.
If your code is mainline, the people who change API and break the compilation are on the hook to fix it.
Either the change is accepted upstream meaning that everyone else making changes to the mainline needs to keep it working, or it isn't and they don't.