Linux kernel maintainers, why are they so grumpy? (2013) [pdf] | Hacker News Reader

Linux kernel maintainers, why are they so grumpy? (2013) [pdf]

elinux.org

66 pointsjacquesm32 comments

Being an open-source maintainer is a mostly thankless job. You put in years of work, and in return you will get endless complaints from random users and drive-by contributors who feel entitled to your time.

I mean, just look at the other thread.[1] The HN mob piles on a kernel developer because of a blogpost where a drive-by contributor severely misrepresents (and misquotes) an interaction he had with him more than a year ago.

When people make comments like "if you treat people like crap, they won't stick around," they're only referring to new or even potential contributors. The maintainers are expected to endure anything that is thrown at them. It's their job, they have to be welcoming. The (non-paying) customer is always right!

[1] - https://news.ycombinator.com/item?id=37671991

To quote some accusations against the maintainer from that thread:

> And some folks wonder why "nerds" and "geeks" are seen as socially inapt...

> If I do kernel sec patches like this, I'll just get a cool URL, a funny image, and publish it as an 0day. That way, I'll get credit AND a cve.

> 100% this. It has all the symptoms of an old boy's club, combined with smugness of socially inept nerds. It would have cost the maintainer basically nothing to handle this better, but the natural inclination of these types of people in their fiefdoms would never allow it.

> If they leave the project, the project won't be losing a good person.

All these people are 100% welcome to use closed-source commercial operating systems if they would like

Related ongoing thread:

I got robbed of my first kernel contribution - https://news.ycombinator.com/item?id=37671991 - Sept 2023 (532 comments)

I got totally flamed for having too-long line lengths on the email list when I submitted my first patch. I wasn't a bit mad, since it was my fault for not reading the contributors docs. I've grown to appreciate the no BS approach they take.

One thing is clear - most developers are really bad at structuring their patches.

I think it makes sense. We spend a lot of time trying to make the code work, and then we're faced with having to spend just as much time submitting the code in a digestible format - most people will get lazy.

I honestly would have to research how to do a patch and I've been a professional dev since the mid-1990s. I guess I would ask ChatGPT?

I've got 100s of my Github pull requests merged into successful OSS projects though.

It is just weird and I guess quaint/old fashioned that Linux requires patches to be sent to an email list.

And yet Linux is the biggest open-source project in existence.

And Git was developed - by Linus Torvalds himself - with email based development in mind from the go.

Yeah, the development workflow is not broken at all. It’s been working great as a means to collaborate on very high quality software. I’m sure a lot of people would prefer it to take place in some Discord group or something. Fml

Emailed patches are not a huge deal for you after submission, and theoretically the first submission should be the same for both (apart from the tool you use to make it). Maintainers reply to emails with comments and you reply to them. The maintainers need to have or setup some specialized tooling to replace GitHub's CI and automation, but you don't care about that[1].

The main difference is that GitHub projects often do squash-on-merge, therefore you add fixes on top and also the commit messages don't mean much. Linux instead keeps the individual pieces so they want you to pay huge attention to the individual commit messages, the buildability and functionality at each step, and squashing each fix in the right patch.

This means that reacting to the reviews, and sending subsequent versions of the changeset, is actually harder than sending the first submission! The reason however is unrelated to the use of an email workflow.

[1] For the specific case of Linux, testing is way out of scope of GitHub/Gitlab's CI; for other projects using email workflow, you can usually push to a branch to have CI run and only submit when it's green.

Many developers these days don’t even know the difference between git and github, which is sad.

There is no denying that linux kernel is a successful project, with serious development, a high quality bar and accountability.

I wish my current project followed that approach. Sadly, it is a bunch of inexperienced developers that think they know it all.

>Many developers these days don’t even know the difference between git and github, which is sad.

This is not sad in my opinion.

GitHub is a whole project management tool and git is just letters management tool.

And what developers need? a project management tool (which includes letters management tool).

GitHub is where the real value is, the git itself is just like 30% of it. Using Git without e.g GH or other "wrapper" kinda sucks.

Wikis, pull requests, code review, issues, roadmaps, CI/CD, community, etc, etc. All of this comes from GH.

_____________

You know what is actually sad? that GitHub doesn't allow to change backend (that letters management engine), so we can switch from e.g Git to other VCS

this makes hard to compete with Git because GitHub only uses Git (afaik)

I think they meant that many newer devs don’t really understand much about the giants whose shoulders they stand on.

But I think that’s always the case with new devs anyway. Takes time to really learn any tool on a deep level

> doesn't allow to change backend

they’ve supported subversion since 2010! https://github.blog/2010-04-01-announcing-svn-support/

The problem with abstraction layers is you are stuck with the worst common denominator of all supported backends. Presumably if you wanted to use a better version control system it would be for its advanced features git lacks. But a system that supported both would have a hard time taking advantage of any feature that was unique to one system.

> I honestly would have to research how to do a patch and I've been a professional dev since the mid-1990s. I guess I would ask ChatGPT?

Because it's hard or because you've never done it?

Because I have never had to do it. All projects I’ve worked on the last decade were on github or gitlab. And prior to that on subversion and mercurial we just would merge branches locally. Back in the 1990s I used sourcesafe which had file locking checkouts and when I used sccs I was the only contributor.

https://www.youtube.com/live/fMeH7wqOwXA?si=7jbkLCm4QalwumNw for a video version of this talk

One funny bit about the talk, which isn’t mentioned in the slides, is that he actually gave the talk at Linaro (the very same company who were called out in the slides as a case study of submitting a particularly bad set of patches)

As an occasional docs janitor and now helping tracking kernel regressions, I also feel "high tension blood" when I see trivial patches from random Chinese contributors, then I reviewed them but these contributors don't reply to my and others's reviews at all. Last year there were people from @cdjrlc.com domain, then this year from @208suo.com. Even some maintainers agree to block such domains [1].

The takeaway is once you submit your patches, it's your responsibility to address reviews that may come to them.

[1]: https://lore.kernel.org/all/5fcffcebca2ef9b8c276e5f4b4464d88... [2]: https://lore.kernel.org/all/ZIoDKqBt2uNSbblB@casper.infradea...

That's interesting, it looks like a reasonably effective way to DDOS the FOSS infrastructure that could be used to mask that something else is being slipped in under the radar. I'd be very suspicious of this, especially if there isn't any response, that means that just getting the mailing lists saturated may well be the primary goal.

I think that's not the point that I want to highlight. The problem, however, is some drive-by contributors have unwillingness to accept feedback (including critical reviews). When that behavior become a pattern within contributor's organization/domain, maintainers may take on the err side. For example, Linux kernel maintainers advise to not run checkpatch script on existing sources ([1], [2]). On a occasion, another maintainer had to roll out comment wording fixes after original submitter ignored the maintainer's review ([3]). And the well-known example is when an university was banned from Linux kernel development [4].

[1]: https://lore.kernel.org/all/20230720134501.01f9f1de@gandalf.... [2]: https://lore.kernel.org/all/2023092730-shampoo-anytime-f437@... [3]: https://lore.kernel.org/all/162b5545-7d24-3cf2-9158-3100ef64... [4]: https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...

[2013]

A substantial amount of the problems listed there are because they were using email for patch management. Better tooling goes a long way to solve those parts.

Won't fix the other ones, but there is no substitute for good tooling.

Related (2016) thread:

Why kernel development still uses email - https://news.ycombinator.com/item?id=12620468

tldr. They're grumpy because most people suck at software engineering and think they're special. The maintainers are forced to continually to deal with wastes of time from people who refuse to read documentation, or are bad at software, and have no real skin in the game.

You forgot employees of Huawei, who send in irrelevant patches because HR gives out rewards to whoever had a contribution accepted and students at the University of Minnesota, who send in crap as part of a faculty research project: https://news.ycombinator.com/item?id=26887670

If only there was a way to reject changes automatically based on some automations...

Who is going to fund it?

You would be grumpy too if you worked thanklessly on an OSS project.

propaganda, given recent drama?

But in the end, he doesn't actually say he will rewrite people's patches, instead he said he would give semi-constructive feedback and give them a status update at some point. Thus it actually seems reasonable.

which is probably what would have happened if the blogposter had submitted to the regular mailing list, but he wanted the extra clout of submitting to the security list, and it backfired

Yeah it seemed to me to be almost entirely about getting credit.

From the kernel mailing list: https://www.mail-archive.com/linuxppc-dev@lists.ozlabs.org/m...

> I will send it to you shortly, but I will switch to my corporate email address.

My guess is that they were trying to get some kudos at work. When they didn't get that, they threw a fit in the form of a blog post.

It's context relevant to a recent popular thread about someone who was upset they weren't acknowledged in the way that they would have preferred.

Reply on news.ycombinator.com