“Please don't waste maintainers' time on your KPI grabbing patches” | Hacker News Reader

“Please don't waste maintainers' time on your KPI grabbing patches”

lkml.org

517 pointsbelter258 comments

I have worked in team with some kernel developers at Samsung, years ago, so let me put some perspective for those who do not understand the dynamics.

In some companies, the amount of patents or Linux kernel patches you get accepted is direct measure of your success.

As you know, whatever you measure becomes a target -- these guys feel very pressed to get ANY kernel commits accepted, no matter how small, irrelevant or nonsensical. Because in the very end almost nobody sees the amount of value that a company creates for Linux users. What everybody is looking at is that "Company X is nth on the list of biggest kernel contributors".

So "KPI grabbing" is meant to mean the practice of not caring for the value or quality of your contributions but rather for the count of the commits or LoC you can get accepted.

This is damaging to Linux kernel developer community because it just creates work for maintainers without creating much or any value at all.

These maintainers are absolutely critical resource for the project and their number and availability translates directly to how much stuff can actually be done. So no wonder people are irked by it.

Maintainers need their own metrics that are effective, and publish them and work to have partners use them.

I like this idea, but the gamification / rules / meta around it sounds like it would be even more work. And like all metrics it would quickly become a target and... Goodhart's law.

This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.

I’m not knowledgeable enough to define what good work looks like but, simply, whenever maintainers see good work, give that user a golden star.

People will work for the carrot, so give carrots for the work maintainers want to see.

Then you get them into trying to get their own people into maintainer positions to let the golden stars flow. Maybe also be more hesitant to give out this starts to the "wrong" people from other companies. Not like there already is enough politic bullshit in open source.

The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue.

But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.

Yes, so either accept the status quo or create positive change.

Given that it’s annoying enough that we have articles and comments on it, perhaps investing maintainers time into developing metrics that work for the maintainers isn’t a terrible idea.

What for? Metrics for metrics sake are a waste of time. The solution is to avoid wasting time on metrics, which is precisely what happened here.

Sometimes, throwing trash into the trash bin is positive change. You don't need to put new trash in as a replacement.

> The issue is that creating objective measure of contribution value is just unfeasible if at all possible.

Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.

It is not about any single patch, you can't assess value of any patch objectively.

For starters there isn't even an objectively established definition of value. You can't objectively measure something that only has subjective definitions.

In a free market you could say that the price could be proxy for the value (ie how much somebody can pay for it should at least theoretically correspond to its value for that person). But how you do that for an open source project?

It might be feasible to assess a single contribution, but not feasible to assess all of them. To make this KPI work, maintainers would have to publish an objective quality rating for each accepted commit. That's an enormous amount of work, and impossible to get right anyway since the value of a change might not be immediately obvious, or might be negated by a future change, etc.

The advice to companies using this KPI would be: let your own dev managers evaluate the quality of the commits of their engineers, and rate them on quality as well as quantity, then submit. Engineers will quickly stop creating busywork when their managers get wise to it.

Doesn't work. "When a measure becomes a target, it ceases to be a good measure."

You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.

Edit: Nvm, I misunderstood the initial problem and my solution doesn't address it.

Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.

Because that would be a huge time sink for the maintainers as well. They’re the only ones qualified to evaluate quality and they have better things to do with the time it would take. This problem is created by these companies and it’s their problem to solve.

I tend to look on commit count as a negative when reviewing open source projects. TeX is probably the highest quality open source project and it's has had maybe 15 commits since I was born. Bash has had about 15 commits in the last five years. Then there's everyone else who's playing to win the Github game with 500 stars and 5,000 commits. That level of activity might impress normal people but it doesn't impress me.

> I tend to look on commit count as a negative when reviewing open source projects.

I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment.

Take the following for example:

https://public-001.gitsense.com/insights/github/repos?p=impa...

by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace.

Software metrics in general isn't the issue, it's the lack of context around them that is the issue.

Disclaimer: I'm the creator of the tool that I'm linking to

Maybe a panel of maintainers could publish a quarterly review with attribution/thanks to the most important contributors. No metrics to game, only pure human expert opinions.

[Generic hand-waving about how LWN kinda does this, in a high-signal manner unfriendly to statistical summation]

Maintainers aren’t interested in taking on extra chores. However, important contributors already do receive attribution, in the format of news articles, here:

https://lwn.net/Archives/

IMO part of the problem is that Greg KH publishes this annual 'state of the kernel' that is just a list of metrics to game. Which feels like it was started to shame non-RH companies to contribute more often. So, kinda maybe beware what you wish for?

Example: https://www.youtube.com/watch?v=SIQr2-Dh0es

Could someone ELI5 to me what KPI even stands for? Does that mean like they're like the shock troopers of open source?

Key Performance Indicator.

https://en.wikipedia.org/wiki/Performance_indicator

Engineer's performance is measured, in part, by number of kernel commits. So, the number of kernel commits is a key performance indicator for the developer.

KPI = Key Performance Indicator = "thing we decided is important for us to measure and increase"

As such exactly what it is, will depend on company/industry/market/team/project/etc. It may be revenue or units sold or widgets made or turnover or customer satisfaction or days without accident etc. In principle, it communicates to teams what is important to business and helps everybody focus and sync.... with usual real-world caveats and implementation risks.

I imagine here, for a large-company linux team, a "KPI" was "how many commits you made" - they measure it, they track it, and reward/recognize/punish team members based on it. So people start working toward increasing the KPI in any feasible way - if you only want me to increase the KPI of commits done, OK, fine, I'll do some commits.

>thing we decided is important for us to measure and increase

This is a great explanation, and I just wanted to nitpick that for some KPI's (e.g. percentage of defective components, or latency), you would aim to "decrease" the KPI, and in some rare instances, you'd want to keep the KPI within a range (e.g. resource utilization, or inventory size, when you want a predefined amount of buffer).

KPI = Key Performance Indicator. It's basically a list of things that are considered during your evaluation. The more items off that check list you cross (if they're one-time items, like "ensure an uptime of X"), or the more of each you do (if they're things like "upstream patches" or "patents granted"), the better your score is.

It's a shit system that does exactly the opposite of what it's supposed to do IMHO, but that kind of stuff doesn't belong in an ELI5 explanation :-D.

KPIs are not a shit system, they are actually important to running any nontrivial organization.

The issue isn't whether KPIs are good or bad but what you do with them.

KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc.

For example, a number of users resigning from our services might be a KPI.

KPIs are an important tool to understand what is going on and whether we are going in the desired direction or not and to communicate this to your teams.

KPIs are a tool for telling your organization what you think is important, to set the incentives in the right direction.

KPIs will not help you if you have no effing idea how to set incentives -- they are just a tool in case you know what you are doing.

Because they are just one step from being a target for your teams, you need to be very careful to communicate what you mean exactly when you set KPIs.

Now, if you make number of users resigning from your services a KPI, you immediately need to set some additional guarantees that this is not overused -- for example by making it difficult for your clients to resign. This might be another KPI (user satisfaction with regards to how easy it was for them to resign).

Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

Humans are extremely good at gaming KPIs, and will do so as long as they're rewarded for it.

Overuse law. If you overuse a law too much it stops being meaningful.

Organizations have hard time improving without measuring their performance and communicating incentives.

There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success.

The best what you can do is compromise, it is unavoidable.

So we know setting targets can make wrong incentives. It is also not a reason to not be setting targets. It is a reason to make sure you damn set those incentives right and take close look that you are getting what you have intended.

The point they're trying to make is, I believe, that metrics shouldn't be targets. In a sense I think that this is obvious to anyone with a rudimentary understanding of applied statistics.

You can't have a (semi-objective) target without a metric.

What you want to measure and what you actually can measure are usually very distinct things, that's where "operationalization" comes in.

You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed.

Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy.

I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.

IMHO to not be gamed, management needs to operate on a conceptual level which is similarly sophisticated, or surpasses in sophistication, the conceptual level of the employees (or suppliers).

If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed.

But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.

As someone who is periodically involved with this sort of thing, the challenge is that far and away the easiest things to objectively measure are almost always output metrics: commits, blog posts, external presentations at conferences, etc.

Output metrics chosen correctly also tend to represent things that the team/person has a reasonable degree of control over. For example, a developer KPI probably shouldn't be number of new customers because that's something they have vanishingly little control over, especially at an individual level.

The problem is that those things that a number of different teams are contributing to are probably the thing that the company cares about.

> The point they're trying to make is, I believe, that metrics shouldn't be targets.

Metrics are either (1) targets or (2) things that you are trying to analyze in relation to the metrics that are targets or (3) a waste of the time you spent gathering them.

The reason metrics are often bad when they become targets is the metrics arr usually not actual direct measures of goals, but things assumed to be convenient proxies, but when you push hard on optimizing them, they stop being good proxies because as well as being easier to measure than the real objective, once you pick any low-hanging productive fruit they are inevitably easier to improve in ways that don’t improve the real objective as much as the proxy (or at all, or which negatively impact it.)

Which would indicate that the issue is not using KPIs to understand how your business is performing, but providing incentives based on KPIs.

This is trite. Just about any way of understanding something is going to create perverse incentives. Keeping the metrics secret has other problems too.

I don't think there is a a good solution. Maybe with radically shortened work hours and less pay disparity, the strives can strive off the job instead.

Actually I'm being silly, there is a good solution: Workplace Democracy. You can fool the higher ups come promotion time, but you can't fool all the people all the time.

Mondragon please get in the tech biz.

Could you please show some real-world examples of this practice?

I am familiar with tech co-ops, but they are small, and so not good evidence.

Didn't Donal Trump fool a lot of people in a democracy? And how does this have anything to do with metrics?

No, he didn't fool us.

yeah right, he didn't fool anyone

When there are just a few decision makers, a standard evaluation procedure is usually necessary for fairness / reproducibility.

When there as many evaluators as evaluatees, no standard evaluation procedure is needed on the theory that all the different ways and biases will cancel out. That doesn't work over time, as clearly America's electorate gets interested in different things, but I am less worried about that for co-ops.

The hope for co-ops vs state democracy is two fold.

Firstly, because people do work many hours at their job, but don't necessarily spend any time running the state, I hope they are more informed and engaged.

Secondly, the "both options are bad, I hate this" problem should be addressed by A, proportional representation (just like should be done with state democracy states), but also B of switching jobs. The barrier of switching jobs is much lower than immigrating, which ought to more than any thing else reign in defeatist apathy. And Co-ops can still fire people, remember.

Um, what? The degree to which a metric is directly tied to incentives absolutely impacts how much that metric will be gamed. It is not trite at all to caution that metrics that are critical to understanding your business should remain as decoupled as possible from incentives. This allows you to maintain the quality of the signals your metrics provide.

This isn't just theory. This is precisely why there are laws that prohibit rewarding people based on how they voted.

So tell me this: how is the degree to which a given metric is tied to incentives meant to be varied?

> This is precisely why there are laws that prohibit rewarding people based on how they voted.

Voting is not an evaluation method of the voter: the voter is the one doing the evaluating. Because the powers should not discriminate between voters as you say, the vote cannot be used to evaluate the voters on and individual basis at all.

Maybe the answer is that: no individual/team assessments of any sort. That makes no incentives. But that also doesn't help with fine-grained strategy.

> So tell me this: how is the degree to which a given metric is tied to incentives meant to be varied?

...by not tieing incentives to it? I feel like I must not understand your question because this seems blindingly obvious...

> Voting is not an evaluation method of the voter: the voter is the one doing the evaluating. Because the powers should not discriminate between voters as you say, the vote cannot be used to evaluate the voters on and individual basis at all.

I don't even get where you got this strawman from.

> no individual/team assessments of any sort. That makes no incentives. But that also doesn't help with fine-grained strategy.

No, just dont use KPIs that are critical to understanding your business in assessments tied to incentices because that will distort your KPIs and make you less capable of understanding your business.

Rather Marilyn Strathern's rephrasing.

The original quote is

"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

> KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc.

Right, but my post, the post I was responding to, the post the post I was responding to was responding to, and the article in the link, was specifically about a situation where KPIs are tied to actual people. I'm not sure what you're arguing for or against.

Measuring things is important, but turning them into a target is a disaster. The term KPI is generally used when the measure is a target.

I have seen many examples of KPI’s not working as intended. Do you have any real world examples of KPI’s that actually worked in practice (and not just in the make-believe heads of managers)?

I didn't know open source programmers were subjected to those kinds of performance reviews. It sounds to me like the pandas have come and visited the penguins and there's a whole lot of shock and horror as the penguins get eaten because you'd think the pandas only want bamboo.

A great many open source programmers are being paid salary and sitting in a corporate cubicle.

KPIs are actually very good when used properly, but they are (like many things) easily misused. For example if you provide a service then your obvious choice for the team's primary KPI would be service uptime.

Use SMART objectives (Specific, Measurable, Achievable, Relevant, Time-bounded) and set a realistic service uptime (e.g. how many nines you want in a given month/quarter/whatever) and KPIs directly enable good decisionmaking at all levels of the org.

Manager up in your grill about a choice you made? Point at the KPIs and say "this will help make it easier to meet our objectives for KPI 1.1" or whatever.

TIL what ELI5 means

Oh god no. It means their managers evaluate them on that metric. In this case, the company has set targets on how many Linux kernel patches employees should land.

It's like the days when outsourcing software development to india was new, and they charged for number of lines of code written.

IBM apparently went that way too https://www.youtube.com/watch?v=kHI7RTKhlz0

This is why ousting Linus puts the long-term health of the kernel at risk. It’s very difficult and expensive, at a personal and professional level, to tell colleagues to “fuck off” if they are submitting low quality garbage for reasons related to their salary. Linux was Linus’s baby, he had nearly absolute control, and he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons.

Now, if the people approving patches are just normal employees who can get fired if a kernel sponsor complains enough, it’s going to be much harder for them to successfully deflect all the bullshit without ever caving in.

I think the email was completely effective in denouncing a behavior. Telling people to "fuck off" isn't required (nor desired, in my opinion).

Linux health happened *despite* Linus' manners, not because of them.

It is naive to claim you can tell which part of Linus is and which isn't contributing to the success.

For one, I think it might be possible Linus's no-nonsense attitude is for the better of the organization as people who can't work with it are leaving causing Linux developers, on average, to be more no-nonsense and also able to coexist better with other no-nonsense people.

But, it is my conjecture only. We would never know how Linux would fare if Linus was another guy.

> It is naive to claim you can tell which part of Linus is and which isn't contributing to the success.

Wasn't the other person doing exactly the same, just on the "pro-asshole" side of the argument?

It is probably easier and more defensible to look at a thing that is, and say "why is it this way" than to see something that isn't and ask, "why isn't it the way I want it to be?"

These are logically not the same structure of argument. I'm not making any character judgements about people involved in this discussion, or on the LKML either for that matter.

It's possible to have a no-nonsense attitude without being offensive.

You can tell people to "go away" politely (yet firmly). It is an art, but it can be done without vulgarity.

>Telling people to "fuck off" isn't required (nor desired, in my opinion).

I'd agree with this for 99% of the time. There's a small 1% of the time though...where sometimes people really need to be told to fuck off.

You can watch a few conference talks where folks really go to town on Linus for his bad behavior.

On GPLv3 lots of calls for Linus to stop supporting GPLv2

"How do we get you to stop..."

https://www.youtube.com/watch?v=PaKIZ7gJlRU

And the list goes on and on. That said, his decisions seem to have worked out amazingly well. At some point I can imagine getting tired at having to repeatedly defend your views.

>his decisions seem to have worked out amazingly well.

As someone who regularly deals with bugs and inconsistencies in Linux, I wouldn't say so. For me personally, I just want to be able to fix the issues, and would rather not spend the time bickering with someone who has some kind of personal vendetta about something that I don't know about. I try hard not to dump my baggage on other open source maintainers, I hope others can do the same.

As others have pointed out, that's speculation.

I have trouble understanding why it bothers people so much that Linus is rude sometimes. You don't have to interact with him if you don't want. You can even contribute to the kernel without interacting with him. Whatever he's been doing has been working for going on 3 decades now, I don't see a burning need to change it because it bothers some outsiders.

There's this weird sense of entitlement. People want to elbow their way into this long-running and successful project and start telling everybody how they ought to behave. They think they have the right to contribute on their own terms without bothering to understand the project culture, and think it's everybody else's responsibility to make things easy for them and behave in a way they approve of.

If you really think the LKML is too toxic and hostile, to the detriment of the project, then feel free to fork it and start up a parallel project without the problems you see. Surely your friendly corporate-style culture will attract more contributors and soon you'll have the more successful kernel, right?

Personally, I like having somebody in charge of the core of the OS who's so concerned with correctness and hygiene that he gets upset when they're violated.

The main problem for me is the old "fish rots from the head down" effect -- when the rude and entitled behavior comes directly from the top, it's no surprise when everyone else starts acting like that and gets at each other's throats. It should be obvious by now where this weird sense of entitlement and refusal to understand other people's culture is coming from.

>feel free to fork it and start up a parallel project without the problems you see

Most Linux distros are basically already doing this. They all have their own patchsets. It's well beyond correctness and hygiene at this point, if you actually look at the changes that are being disputed, it already falls a lot more in the "cultural differences" category.

I think the kernel developers got on pretty well for all those years and managed to put a pretty good product together, despite the big rotten head. And Linux distro maintain patchsets, but they're against mainline, that's a very salient difference here. They still start from whatever upstream publishes, they're not so upset by swearwords that they took a snapshot at Linux 2.4 and only add their own code on top of that.

The point there is, while some negative attitudes may not be preventing other projects from using the code, they do prevent the project from growing, and it leads to fragmentation, which I believe they are seeking to minimize. So something has to give.

That's fair and I don't disagree. Personally I never tell anyone to fuck off, but humility prevents me from assuming Linus is wrong for doing it.

So the LKML has been rotting from the head down for...30 years? If the result of that is Linux, maybe the rot isn't as bad as you fear.

The cultural differences that cause distro forks are debates over free-vs-libre, what belongs in the kernel vs userspace, and standardization issues, not (afaik) "we won't submit this to the upstream kernel because we don't approve of the way they talk to each other there". And I think it's very important for the core kernel devs to hold the line on those issues.

> I have trouble understanding why it bothers people so much that Linus is rude sometimes.

Can people actually point to instances where he was rude?

Well...I like Linus just fine as kernel maintainer, but I remember a dozen times (over 2 decades, to be fair) when he was, uhh, very direct. It's often not as aggressive as people make out, but he can definitely get a bit mean when people keep making the same mistakes.

If anyone gets ousted from the process, they probably deserve it, for wasting more maintainer time. Nobody is immune from that, not you, not me, not the maintainers, not the BDFL. Expecting the BDFL to be able to have full control and to review every patch is also nonsense that causes the exact same type of issues: https://lwn.net/Articles/393694/

Good theory, but it's factually wrong. "Maintainers" in this context doesn't refer to Linus personally, it refers to individual maintainers of parts of the kernel. You can check the kernel MAINTAINERS file, and note that such a file has been there for years.

There has been a longstanding problem of people submitting low-quality patches to maintainers - not to Linus personally - and that problem predated Linus stepping back for one release.

Linus was not ousted and continues to have nearly absolute control; he has, for decades, chosen to exercise that control by delegating ownership of parts of the code to other people. Those people receive and review patches and incorporate them into their repos, which Linus pulls in bulk. (This is the origin of the term "pull request," which predates GitHub's use of it.) He quickly reviews those patches to make sure there's nothing weird, but he generally trusts that his "lieutenants" are making good decisions.

That means that the time being wasted here is the time of individual maintainers, who are reviewing the patch, making a full judgment on it, and pulling it into their tree. Linus sees all such cleanups once per cycle and looks at them fairly quickly.

Almost all of those maintainers have been "normal employees" of various companies for many, many years. Again, take a look at the MAINTAINERS file.

> he didn’t care too much about politeness - this was a magic recipe for him to be able to stave off garbage patch requests from motivated corporate peons.

Linus disagrees with you, so strongly that he invested a great amount of his time, reputation, and political capital in changing his behavior and that of his organization.

https://news.ycombinator.com/item?id=18000698

I think I'll trust Linus on that. Linux seems pretty successful.

Their younger versions might not have agreed with it, but some people do mellow out in their old age. In this case, to me it's clear that he is doing a large turnaround, which validates Wyager's statement that he did not care too much before.

I'm not sure what that means, or what basis there is for any of it ? The basis for my comment was Torvalds' own statements and actions, and he's a pretty good source for Torvalds and for effective management and communication in the Linux community. Do you know him personally? Are you a maintainer?

It means that Linus used to be fairly brusk when he was younger (basis: e.g. flipping people off and telling people to "fuck off"), and recently he has apologized for his behavior (basis: the link you shared). To me this is him mellowing out in his old age. His younger self might not have agreed with this mellowing out, seeing it as unnecessary (basis: his responses previous to the apology to other people who called him out on his behavior). The fact that he apologized means that even he realized that he did not care too much before, but now he does, and therefore he wanted to make up for what he has done.

> To me ...

My question remains: What is the basis of that? Do you work with him? Read his biography at least (assuming there is one)?

The basis is the sentence before that one.

I guess it's just Internet speculation. I am looking for something with a real basis (not that you are required to give it to me).

Thanks for your perspective I didn't actually realize that dynamic was at play (at an organizational level, anyway, I'm well aware individuals game # of contributions).

These organizations pushing trivial/garbage patches and wasting maintainers' time absolutely have to be called out by the community as unacceptable, so I'm glad to see the post. This is a massive waste of time, time which open source contributors/maintainers often are already short on.

Sounds like the typical issue of business intelligence using quantity over quality to rate work performance

"When a measure becomes a target, it ceases to be a good measure."

Corollary: "Businesses that base their growth on A/B testing only, turn into gambling or pornography".

Relevant xkcd: "Video Content" https://xkcd.com/1804/

A/B and telemetry-worship reminds me of something I posted on Fedi earlier. Pasting it below:

---

"Guys, I have an idea. Let's make a browser optimized for the kind of people who don't opt out of telemetry, and market it to privacy-conscious people." -- Top minds at Mozilla Corp

If you market to privacy-conscious people and you base all your decisions on data from telemetry, you're gonna get a very skewed perspective. You'll see disproportionate representation from fans (people who want to enable telemetry because they love Mozilla and want to help, and will rationalize any decision Mozilla makes) and very non-technical users who don't know what telemetry is and how opt-in/out works. Technical users who disable stuff like telemetry, analytics, etc. are going to look like a rounding error.

This explains a lot of Firefox's changes.

This is enlightening and depressing. It's sad to see the Fox let itself go.

I sometimes wonder if something like this has happened with money.

The committer defends his patch, and Qu responds very constructively with this list of more important work to tackle https://lkml.org/lkml/2021/6/21/342

Qu sounds incredibly awesome.

Can we measure "impact" of changes?

PKI measuring just commit count is pretty silly. Like SLOC. Cite folklore.org story of -10,000 lines added.

Whereas Qu's suggestions are useful, important. Am totally ignorant about Linux, kernel, etc. But maybe there's already a leaderboard where the hivemind helps prioritize work.

A measure of the ratio of patch effort to review effort (as subjectively judged by the reviewer, perhaps to the nearest order of magnitude) might be informative.

Belated reply. Sorry. I really like this notion. But I can't imagine what it'd look like. Please post anything you come up with. Thanks.

That was a shockingly productive and constructive resolution.

Are we doing Hacker News wrong? Should we have a system to encourage models of good and productive interchanges, which leave the participants happy and the product better? I think hostility is easier but I guess it's also addictive, even for us.

I don't want to be shocked by seeing this. I want to see more of this.

Thanks for posting this. I had initially wondered how, if not through this process, you were supposed to fix minor stuff like comments and error messages.

>I'm not saying cleanup is not important, in fact we have routinely cleanups of typos/grammar for btrfs. (And I guess mostly caused by myself?)

>Please at least merge all those small fixes into a larger patchset, and with a good cover letter to explain the reason (and auto-tool to do the change if possible) for all the involved maintainers, so that all of us are on the same page.

The fellow who is being named in this particular PR has been _quite_ busy lately, submitting mostly typo fixes and whitespace fixes: https://lore.kernel.org/lkml/?q=f%3Athunder.leizhen

Sometimes their patches aren't even valid - they tried to fix "borken" to "broken" and the maintainer was not happy: https://lore.kernel.org/lkml/YK3wOkX6I78j73zD@gmail.com/ (this does come down to not being familiar with this particular bit of slang - but they push back and argue a bit which doesn't help)

Sometimes the maintainers are not happy just getting a patch that fixes one line of whitespace: https://lore.kernel.org/lkml/20210608105943.2376328c@oasis.l...

Here is this person's commit history for the last year for additional context:

https://public-001.gitsense.com/insights/github/repos?q=auth...

They do appear to contribute regularly enough (relatively speaking) and based on some of the busfactor metrics, they are the sole maintainer or main maintainers for about 50 files.

And if you look at their one line change commits, they do seem to be valid:

https://public-001.gitsense.com/insights/github/repos?p=comm...

Disclaimer: I'm the creator of the tool for the links above

> And if you look at their one line change commits, they do seem to be valid

By construction, yes. You're only looking at their commits that were merged into Linus's tree.

Yes you are right. I was more establishing this person's successful contributions and that they appear to provide meaningful contributions. What is happening now with all the new proposed changes that is causing a stir is something that I really can't speak too since there is no easy way to navigate their proposed changes.

> they tried to fix "borken" to "broken" and the maintainer was not happy […] this does come down to not being familiar with this particular bit of slang - but they push back and argue a bit which doesn't help

I did not know "borken" either, but I am aware of "borked" and "broken". Based on that email thread someone else already attempted to fix this in the past.

Maybe it's an indication that the so-called "joke" is not actually funny and it should be adjusted to either "borked" or "broken" to not cause others to send the same fix?

Agreed. While the kernel developers can obviously do whatever they want in terms of in-house jokes, the claim that "this is just normal usage, Google it" is a bit ridiculous. They could have just said, "we know it's wrong, we just thought it's funny".

("Borken" is a city in Germany, that's my only association with it )

> ("Borken" is a city in Germany, that's my only association with it )

Being German myself it's also the plural of "Borke" (i.e. the "bark" of a tree): https://de.wikipedia.org/wiki/Borke

I can't remember every saying or reading / hearing anything that involved a multitude of barks though, hence why, having lived in NRW, that association is still more prominent in my mind.

They could also say "this is just a trap to waste the time of people who want to increase their Kernel commit counts. try again".

This happened quite often in openstack too. Since any contributor with a merged change a few months before the annual conference got a free entry ticket, you could see people doing trivial formatting fixes at a certain time of the year.

Similar to the issue with Digital Ocean and Hacktoberfest https://blog.domenic.me/hacktoberfest/

People are always going to try to game metrics.

from an Ironic core.

There were several months where I think we had contributors running a spell checker against patches as a review. They would miss obvious code syntax issues but would request lots of spelling changes (including British->American spelling).

You get what you measure. There's always going to be someone trying to do the absolute minimum to increment the number they get judged on.

I see that these "cleanup" patches are not bringing much value, bust I also don't see why fixing spelling mistakes or log messages is considered as harmful, KPI boosting or not.

The maintainer even said if someone else sent those patches it would be OK, but not if Huawei employees do it. If they distrust Huawei so much, why not just ban them from committing, the same way they did recently with university "security researchers".

I guess if you have hundreds of trivial patches it'd take ages to review and merge them, and due to kpis Huawei are being deliberately inefficient

Is this an issue with the workflow that the kernel uses in terms of time spent per patch? In most workflows this seems like it'd take under a minute to review, approve and merge each one although the CI/CD might add some computing cost.

It takes more time because they're not just being rubber stamped.

These small patches create some overhead for maintainers. He actually said in the follow up message that grouping them together is fine.

Please at least merge all those small fixes into a larger patchset, and with a good cover letter to explain the reason (and auto-tool to do the change if possible) for all the involved maintainers, so that all of us are on the same page.

I understand the overhead, but it seems like bundling lots of unrelated cleanup fixes into one large commit would make it easier to sneak in something nefarious.

They’re still going to be reviewed, and if the cover letter says “typo fixes” and one of the patches fixes more than a typo, that would be an instant red flag. A single bigger patch set is faster and easier to review.

I think the general idea would be something like, "Fix spelling errors in log messages" and bundle several changes that do that.

All parts of the patch have to be reviewed regardless. It just saves one session startup/tear down per patch.

I'd argue it would make it LESS likely; seeing a hundred "cleanup" merge requests that are all basically the same thing vs a single one where something more than just cleanup would stand out.

My understanding is that they're not necessarily asking for the commits to be squashed, they're asking for them to be submitted together as a patch series.

Patch series are an important part of a healthy review workflow because they allow both a micro and a macro view of changes. I have written about this before: http://nhaehnle.blogspot.com/2020/06/they-want-to-be-small-t...

Right? I am not really sure I see the problem. It seems to me like Huawei is just submitting clean-up patches and the maintainer has higher expectations of what patches they should be submitting. Granted I am not a developer not a maintainer of any project so that might be why it is not obvious.

If there’s a pattern of them submitting a ton of these kind of patches, that could be viewed as gaming a metric. Huawei might, for example, be assessing kernel developer performance partially on # of upstreamed commits, and I guess that’s what is being suggested here.

Doing as many patches as you can may make the approvers fatigued and more likely to approve something that is bad or malicious.

It's not about security. It's about wasting reviewers' time. I too would be annoyed if someone was submitting, say, whitespace-only changes, or similar cleanup, non-functional changes - I still have to review them and my time would be better spent looking at actually meaningful changes that make the product better. It's OK if someone is just learning the ropes with an easy change, but this seems to be more like people trying to game a metric by submitting lots of small changes, which wastes the reviewers' time for no actual user benefit.

I think the proper way to do a “cleanup” patchset is to communicate with the maintainer beforehand and ask them what they’d prefer

Or as the maintainer themselves have indicated, bundling a number of these changes together along with a cover letter explaining them.

Would you rather spend 5 minutes to read a cover letter and code, or just read the diff see word_a has been changed to word_b in comments? Trivial typo or cosmetic changes should stay small, and can be approved in a few seconds.

I think there was some medium post by a guru where he claimed the best way to get experience was to submit as many spelling error pull request as possible. That way he can throw in his resume that he's contributed to a dozen or so open source projects. Given how broken the hiring process is, it wouldn't surprise me if this works

Ugh. Next thing we’ll see is a LinkedIn widget on your profile page showing how many projects you’ve committed to.

I know GitHub does something like this already, but GitHub is more techie sharing, and LinkedIn is more about bragging based o what I see.

That sounds more like fabricating experience than getting experience.

There is a certain balance to this however. Where I work we generally have a rule of no formatting changes mixed in with real code changes to make reviewing easier. It's a huge pain to see a ton of white space fixes mixed in with logic changes. As long as they're separate commits in the same pull request it's usually OK. We also generally don't have formatting-only pull requests because it just adds overhead and indirection to git-blame, and doesn't add value to the end product.

It's likely the context of available time and resources that matters here. Huawei has the ability to contribute a lot, but instead sends an experienced, highly paid person around to hammer in a nail that was sticking out slightly and calls you over (when you could be doing something more productive) to high five a job well done. This would be appropriate for an beginner apprentice.

Yeah, it might've been slightly annoying to someone, but it's a visible waste of time overall.

When people make trivial typo PRs on my open source projects I say thanks very much, close their PR, and make the change in a commit under my name. This way, if their interest is in improving the project then it worked. And if their interest is in contributions then it works also because it doesn't pollute their contribution history with trivial changes that aren't real contributions.

That’s incredibly disrespectful. With that approach, I would be shocked if anyone would want to contribute to your project again after such behavior.

Maybe it's easy to feel it's disrespectful because they've made social networks out of contributing to repositories.

Accreditation for the modification of petty and trivial issues is a nice and decent gesture, but overall unworthy of a paper trail binding me to any random none-of-my-business project for such a drive-by PR.

Personally, I would feel more comfortable sending that kind of inane PRs if the parent's way of merging was the standard one, but I know it isn't and therefore opt to keep separate accounts to compartmentalize (which unfortunately increases friction to such contributions).

> Accreditation for the modification of petty and trivial issues is a nice and decent gesture, but overall unworthy of a paper trail binding me to any random none-of-my-business project for such a drive-by PR.

I don't know about others, but before investing significant time in a project, my first action will be to read through the documentation, then the code (and accompanying comments), and then I'll make minor fixes along the way.

After this, I'll submit a PR with my changes. If a maintainer were to close my patch and then proceed to commit the exact same changes under their own name, I'd never contribute, use, or improve that project again. Why would I waste my time?

That kind of behavior displays an immense lack of respect for contributors.

I was referring to a PR which literally changed a single letter :) Not a meaningful piece of proof-reading work.

Sorry, you missed my point. I'm talking about people who tour around GitHub making PRs comprising a single letter typo charge in order to bolster their contribution statistics. For example the last one I got was changing Github to GitHub in one location. I have no time for that whatsoever. I'm otherwise an extremely friendly and respectful maintainer, including with beginners, of course, and my project has various repeat contributors; I'm sorry I can't point you to it to convince you of that. Your comment is a misjudgment.

I think the concern here is that at huawei there's some kernel patch metric, or incentive, or something like that. And following Goodhart's law... "When a measure becomes a target, it ceases to be a good measure."

So now there's a target that costs maintainers time, possibly lots of time, and costs huawei nothing.

There's really no control here to dial it back other than to ask 'please don't do this'.

>> I see that these "cleanup" patches are not bringing much value, bust I also don't see why fixing spelling mistakes or log messages is considered as harmful, KPI boosting or not.

My best guess: It creates a small amount of work for a maintainer to review and merge. This is worthwhile if a new contributor is learning their way around, and getting new contributors is very important to an OSS project. To have a large company submitting a bunch of these is a distraction for maintainers who have better things to do than support someone trying to inflate their metrics.

It's true that it creates an overhead for maintainers. But, on the other hand, maybe Linux process needs to be improved to relief the maintainers from having to review every single patch.

Linux as a project is big enough, and if maintainers are the bottleneck, maybe it's time to have sub-maintainers to whom such tasks could be delegated.

See, this will just cause additional communication overhead for the maintainers, too. And for what benefit, just to accommodate the behavior we see here?

> maybe it's time to have sub-maintainers to whom such tasks could be delegated

They already have subsystem maintainers. The result was an even faster process.

https://youtu.be/fMeH7wqOwXA

why? to support whatever arbitrary demands third parties might make on their time?

I think insisting on quality is way more reasonable.

The easy solution would seem to be for the person accepting the patch to check a box that says 'this patch materially improved the codebase'. That would put an instant stop to gaming the KPIs.

In general fixing spelling or logging errors are P4 or P5 at best so they should only be fixed if something more important is touching those files. That how I've always viewed it.

Looks to me like the problem is not the content of the patches, but the way they are submitted as many small patches all from the company. I suspect the maintainers wouldn't be as concerned if all of the contributes were independent (no "@huawei.com" address) or if they bundled the patches together to reduce overhead.

I would agree with you. Those commits would be the easiest one to merge. How difficult is it to approve a typo correction? Saying a typo correction is not productive is actually the opposite: it takes very little time to review and approve. I don't know why people are mad that these commits are not from a student.

The dialog between the author of this message the responses on the mailing list add some clarity. There's nothing inherently wrong with providing cleanup fixes to this kind of project, but submitting those as individual merge requests instead of one grouped patch is just serving to increment the developer's "number of patches submitted" KPI at the expense of the time spent by maintainers reviewing every individual patch.

The sense I got from the post was: "It's fine for noobs to cut their teeth on small issues, but as one of the largest tech companies, I'd expect more from your commits. It's obvious you're not even trying".

I agree with this. It's reasonable to expect more from well-paid professionals.

The point of the email is that the submitter is indeed trying, trying to improve their stats within their organization, at the expense of maintainers’ time, rather than working on the kernel in good faith.

Open Source is such a surprisingly high trust ecosystem. I really hope that isn't something that has to be changed in the future.

I can 100% understand the concern here.

Possibly there is some sort of internal corporate metric or just personal bragging rights get tied to these kinds of patches. It costs the company nothing to incentivize this and the maintainers get all the work associated with it.

Like all metrics "When a measure becomes a target, it ceases to be a good measure." (Goodhart's law). There's really no reason for huawei to change other than for the maintainers to ask them to not do that thing.

I recall a talk from a maintainer once gave at some company (I forget where). He detailed what kind of patches get accepted and what don't. They were pretty straight forward standards, some more obvious than others (you know ... say what the patch does accurately).

It wasn't stated but he clearly expected some level of professionalism or just efficiency from the folks giving him work to look through their code on behalf of their company.

I don't think that's too much to expect.

There's an external metric too -- there are public lists of "Biggest company contributions to Linux" where the likes of Intel/Google/Red Hat/NVIDIA appear:

https://news.itsfoss.com/huawei-kernel-contribution/

It's good for PR.

Also from the thread posted earlier:

> I'm not saying cleanup is not important, in fact we have routinely cleanups of typos/grammar for btrfs. (And I guess mostly caused by myself?)

> Please at least merge all those small fixes into a larger patchset, and with a good cover letter to explain the reason (and auto-tool to do the change if possible) for all the involved maintainers, so that all of us are on the same page.

Discussion in Chinese online community zhihu if anyone is interested:

https://www.zhihu.com/question/466111598/answer/1951896502

To save English speakers a couple of clicks to translate: https://translate.google.com/translate?hl=en&sl=zh-CN&u=http...

Keep in mind this community is probably censored:

https://qz.com/1063073/in-china-you-now-have-to-provide-your...

From the online translation, the mention by one the commentators that the Kernel Maintainer is also Chinese...give me a bit of a chill...

To make it clear ...Not that I would not trust the maintainer, instead, my concern is that depending where he is based a window could be open to unpleasant pressure...

Yeah real identity has been going on for a while and recently I had to submit my id number too but I don't think it affects most of the discussion on Zhihu as long as it's not too politically sensitive. It's an annoyance for many people in oversea though.

As I am not a Chinese speaker..do you confirm there is in the thread a comment that the maintainer is also Chinese ?

I am concerned about possible pressure from Huawei and its "shareholders": https://www.bbc.com/news/business-53172057 to a Kernel maintainer. Looking at the heavy down votes my post got its not a concern here...

Yeah one reply says so. The name is Qu Wenruo (definitely a Chinese name) and his github repo is at: https://github.com/adam900710

From the email it looks like he works/worked in SUSE.

Your post are downvoted because it is borderline racist.

The maintainer happened to be Chinese and that gave you 'chills'. There are 1.4 billion Chinese and most of them live in China, there are also open source projects hosted in China, and a lot of open source contribution coming from people living in China, if only because there are many people there. The media links you posted are also made of conjecture ('Trump administration claims' is literally in the title) and you've then stretched them with your own imagination to arrive at your 'concerns'.

The comment is not racist because if you did read it the concern was NOT about the maintainer being Chinese and I made that VERY CLEAR.

The concern was that in the Chinese group discussion commentators were mentioning he was Chinese. Now why would that be relevant indeed ?

It can be relevant because it open the possibility to him being pressured by a company that is partly owned by the Chinese Army. That is where the chill was coming from... Hopefully he is based in the US.

Now tell me, where is the racism ?

Who Owns Huawei ?

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3372669

Who Owns Huawei? The Company Tried to Explain. It Got Complicated.

https://www.nytimes.com/2019/04/25/technology/who-owns-huawe...

This is a detailed refutation of that paper: https://pekingnology.substack.com/p/who-really-owns-huawei-a...

It goes into the specifics of the Chinese laws governing employee share ownership, and why the employee shares are held by the union (it has to do with arcana of 1990s-era Chinese laws governing stock distribution by non-publicly-traded companies). Balding & Clarke fundamentally misunderstand why the union is used as a share-holding vehicle, and what the legal implications of that are.

The majority of the stock is held within 华为投资控股有限公司工会委员会. Of course one would be naive to believe that the _power_ is held within the same instituition too.

All of that hoop you jumped through is (seen from my perspective) thinly veiled xenophobia/racism. Now it maybe that you did not intend it that way, but it came across as such, for instance you've posted:

> partly owned by the Chinese Army

None of the sources that you've posted supported this, The PLA is also specifically banned to operate/control business since 1998. The ownership structure of Huawei is also available in more detail here: https://en.wikipedia.org/wiki/Huawei#Ownership

If it's not some deep seated bias, what lead you to straight up invent or stretch claims like this?

=========== DiogenesKynikos 7 minutes ago [–]

This is a detailed refutation of that paper: https://pekingnology.substack.com/p/who-really-owns-huawei-a... It goes into the specifics of the Chinese laws governing employee share ownership, and why the employee shares are held by the union (it has to do with arcana of 1990s-era Chinese laws governing stock distribution by non-publicly-traded companies). Balding & Clarke fundamentally misunderstand why the union is used as a share-holding vehicle, and what the legal implications of that are. ===========

https://en.wikipedia.org/wiki/Labor_relations_in_China#All-C...

"Independent unions are illegal in China with only the All-China Federation of Trade Unions permitted to operate."

"The Trade Union Law in China forces the all trade unions in China dependent with the Communist Party of China."

The article that I linked goes into great detail about what the legal relationship between the employees and the union committee is, what level of control the union committee actually has over the shares, what happens if the company dissolves, and so on. Just saying that the union is a member of the national federation, and that the company is therefore owned by the Communist Party, is grossly inaccurate.

The bottom line is that the union committee acts as a legal vehicle that allows more than 100 employees to hold shares, and that it can't just do whatever it likes with the shares, because of its legal responsibilities to the employees. This structure was used because decades ago, it was not possible (or at least legally tricky) for large private companies in China to issue shares to their employees.

"What’s the Deal with Huawei’s Ownership?" https://sayari.com/resources/huaweis-ownership-opaque-unusua...

"Huawei claims to be owned entirely by its employees, but verifying this claim is a tricky proposition...However, comparing Huawei to other Chinese corporations reveals that Huawei’s ownership is highly unusual." .... "Using Chinese corporate data, we determined that Huawei’s ownership structure is very uncommon, particularly for a company of its size. Fewer than three hundredths of one percent (0.0249%) of active Chinese corporations in our dataset are owned by trade union committees, like Huawei."

"...We set out to answer this question using Sayari’s database of 55 million Chinese corporate entities... ...Of these 55 million Chinese corporate entities, approximately 38 million are active."

"...But ownership by a “labor union committee” (工会委员会), like the Huawei case, is very rare. We identified just 4,547 active Chinese corporate entities that list a labor union committee as a shareholder. This represents just 0.0249% of the active non-household enterprise companies in our dataset. Among companies operating in the technology sector, ownership by a labor union committee is even more rare. Only 794 active technology companies in our database—or 0.0201%—are owned directly by a labor union committee..."

"...Of these 794 companies, Sayari was able to identify the registered capital in Chinese Renminbi for 526. Of these, just three had registered capital greater than 1 billion RMB (approx. USD $148 million); Huawei has registered capital of more than 16 billion RMB. Chinese companies of this size usually are major state-owned enterprises, publicly-traded, or both. In other words, it is practically unheard of in China for a labor union committee to own a company of Huawei’s size, scope, and global reach..."

The article I linked to explains the origins of Huawei's ownership structure. It made sense when it was set up decades ago, though it's not the way that a company would be set up now. Yes, it may be unusual for a company this size to be set up this way, but there are very few companies of this size anyways. There does not appear to be anything nefarious about the use of the labor union committee as a vehicle for employee share ownership. It just appears to have been a reasonable way to set things up when Huawei was starting out.

Not really:

"Who Controls Huawei?" https://www.ui.se/globalassets/butiken/ui-paper/2020/ui-pape...

"Technical dependency and the issue of party-state control over Huawei"

"...While at first glance Huawei’s line of defence appears convincing, there are also reasons for doubt. It is true that the company is privately owned by its employees, but there is reason to believe that ownership does not come with control..."

"...Huawei has not only profited from party-state support, but is operating in a specific political, legal and economic environment that makes it impossible for the company to be fully independent..."

"....The widespread distinction between privately owned enterprises (POEs) and state ownership is not decisive in China. It is not ownership but the issue of “state capture”that is pivotal... "

"...Following an official request, Huawei has confirmed to the author of this paper that party organisations exist within the company in accordance with Chinese law. The company has provided assurances that party organisations have no influence on the company’s business operations and that it is run by an independent management team. The party organisations are mainly responsible for educating employees. The general findings on the deep linkages between the economic and political elites, however, call into question whether such differentiation is adequate in the context of China’s political economy..."

"...China lacks an independent judiciary. While theCCP government speaks of strengthening the role of law in its governance, it follows a rule by law principle and does not intend to implement the rule of law. The CCP has an essentially instrumentalist understanding of law. Law is seen not as a constraint on power, but as a means of power. This means that legal certainty does not exist in areas of crucial political importance. Hence, laws do not provide reassurance against political interference in the business operations of Chinese companies..."

"...China’s Intelligence Law,raise even further doubts about whether the law could shield private sector companies from political interference. Article 7 of the Intelligence Law enacted in 2017 and amended in 2018 requires any organisation and citizen to “support, assist in and cooperate in national intelligence work”..."

"...Scepticism over Huawei’s independence from party-state control does not just stem from the political, legal and economic environment in which it operates....Huawei’s founder, Ren Zhengfei, is widely believed to have a past in China’s People’s Liberation Army. His daughter and the company’s Chief Financial Officer, Meng Wanzhou, is suspected to have held a public affairs passport.... A widely discussed and controversial analysis of the CVs of leading Huawei engineers found a large overlap with China’s security apparatus..."

"...This clearly demonstrates that the main control over Huawei does not lie with the employees of the company, who can only choose 83 people from a preselected list of 109 candidates, but with the nomination teams that hand-pick these people using vague criteria. Whoever wants to control Huawei needs to control the nomination teams. There are no adequate checks in place to ensure that it is not the Chinese Communist Party and its organisations within the company that are the ones exercising such control...."

"...What sparks further suspicion of the process is that none of the Huawei employees I have talked to was aware of the nomination process, even though it turns out that real power of selection lies with the nomination teams..."

"...In the light of these four factors – the general level of independence of the POEs, the legal environment and Intelligence Law, state support for Huawei and the company’s governance structure – it is very difficult to reject the claims of Huawei critics that the company is more than just a privately run company striving for economic profit..."

Not really what?

The article you linked acknowledges the legal reasons that Huawei's employee share ownership was set up the way it was (limits on the number of individuals that private Chinese companies were legally allowed to distribute stares to). Most of the concerns the author has with Huawei are really just general concerns about any company in China - namely that the state theoretically could exert influence over them.

Its about who controls the company not who is on the record as owning. Did you read the part about the selection committees ?

> commentators were mentioning he was Chinese. Now why would that be relevant indeed ?

Because the maintainer could've been a foreigner who sees a "huawei.org" email address and imagines that they might want to apply unpleasant pressure to get their typo fixes merged.

Having a Chinese name makes it a bit less likely that the maintainer is influenced by such stereotypes and makes the critique of Huawei harder to dismiss out of hand.

The real question is how likely is it for this mainland Chinese maintainer to have allegiances to the Chinese regime. Actually, even when holding convenience passports, the question still holds [0]. And it wouldn't be the first time Huawei employees tried questionable things. [1] [2]

[0] https://www.justice.gov/usao-az/pr/former-raytheon-engineer-...

[1] https://globalnews.ca/news/7275588/inside-the-chinese-milita...

[2] https://blogs.cisco.com/news/huawei-and-ciscos-source-code-c...

BTW, mentioning his origins probably has something to do with "saving face", it was probably more embarassing to the Chinese patch submitter to have been criticized by another Chinese person compared to a Westerner, since in Asia the mentality of "we're from the same country, we'll look out for each other, and one of us shouldn't be making another look bad in public." is more pevalent.

>进git看了一下，笑死了。这位华为大佬曾经在一天里对同一个文件提交了6个fix，每一个fix修改注释里面的一个拼写错误，还有一个是调整include 顺序。被reject了以后还发邮件去defend。打个比方就是你假装帮导师做数据，一个数据没做但是一天发给他6个版本，每个版本改论文里的一个错别字，同时要求导师把你名字加到作者里。导师说求你别给我发了，然后你跑去办公室和他争论这个错字很重要，换了谁都要发飙啊。

https://www.zhihu.com/question/466111598/answer/1953367097

this comment is funny.

Yeah some comments are funny and insightful, but again I have no idea whether what they said was true or not.

One of the answers raises an interesting possibility:

https://www.zhihu.com/question/466111598/answer/1953785089

Basically, they claim it's possible that it's not about KPI at all; but rather, Huawei has added some internal processes to try to raise code quality. But those processes, rather than focusing on things like algorithms, ends up mostly catching unimportant things like minor security issues in tests and code style issues in third-party code. So they suspect that these trivial patches are somebody internal to Huawei trying to satisfy their own internal processes by fixing upstream.

That would mean it's still about KPIs, just not those of the patch submitter.

Got hit a while back with a GitHub account that was "bombing" OSS projects with Docker image "best practice" fixes related to Deb "recommends" or some other such stuff. Most of their history for the past year was exclusively opening these PRs with canned messages.

Some of the projects were very large(Apache Arrow, a large Google project IIRC), and they were opening commits up against largely test images.. And a LOT of them. On one of the larger projects somebody rightfully asked "What's the value add here?" who was promptly ignored by what I can only assume is a project manager who start humoring the submitter by trying to help them get this entirely useless work merged. It turned into an epic fail-whale causing all the projects tests to fail and that initial dev spent "too much time" trying to untangle it all before putting their foot down and reverted the changes.

Other project maintainers got wise to these fishy PRs and after inspecting the submitters history just started telling them to sod off.

What does "KPI" mean in this context? Never heard the term before

Key Performance Indicator. The implication is that some Huawei engineers are rewarded for contributing to the kernel.

Key Performance Indicator: ie, someone at huawei’s job performance review depends on “made N open source contributions to major projects”, and Goodhart’s Law strikes again.

Came to the comments to learn what KPI was. And leaned about Goodharts law: https://en.m.wikipedia.org/wiki/Goodhart%27s_law

You are one of today's lucky 10,000! https://xkcd.com/1053/

Saving someone a click

> When a measure becomes a target, it ceases to be a good measure

And ironically, earned 3 points of HN karma kreds for this very tiny contribution to the discussion. Oh the deep irony in this particular context of KPI spamming.

HN posts/karma are probably going to come up in his next performance review!

key performance indicator? kernel programming interface?

I too ponder the meaning.

"Key Performance Indicator". I believe the implication here is that some people from Huawei are sending useless patches just to pad their performance values and look better inside the company.

Indeed... although I'd point the blame more squarely at management deciding that "number of patches" is the right way to measure someone's performance.

Not that I have any idea if that's the case here, but I've worked in places like that. Idiotic approach.

Is there anything going on here that suggests this is not a routine maintenance issue?

The Kernel maintainers have a lot of people who want to submit code to the kernel, and have to maintain standards. I'd imagine there are a lot of emails like this one.

Good on Huawei if they are incentivising open source contributions. Shame on them if they are wasting OSS maintainer time. Presumably well done Qu for promoting some order in a chaotic world.

It's not nefarious, it's a pattern of a particular company trying to game a metric, to boost their reputation.

Pretend you were a dev, and your effectiveness was mostly described within your 500-engineer department in terms of "how many pull-requests" you merged each month. In the abstract, that seems like a reasonable metric to use - it does seem to correlate with actual output, and encourages smaller slicing (which is generally positive). But two years down the line, you find this one guy, consistently in the top 10 rankings, and you feel like he's just not accomplishing that much, so you start looking into his actual workstream. And you find that he does do normal work, but he produces 10 times as many 1-5 line PRs as the average, each of them adjusting some bit of copy or changing an error message. For the better! And his QA isn't bothered, because his work is very easy to verify.

But he's not actually one of the most effective devs in the company, and it's really irritating that his process-games (a) seem to work (the VP of engineering definitely knows him as a 'high output engineer') and (b) impose some unnecessary load on other bits of the company's process (like the QAs and burning CI credits).

That's the emotion in play here. "Stop wasting our time, we can see what you're doing."

This looks boring. This is a scene that plays out in every company weekly. The stakes are trivial, there is no evidence of ill intent anywhere. Nobody has done anything particularly badly behaved. Nothing here is a shining example of good behaviour. There isn't a feel-good aspect.

I suppose the question is, who are the 100+ people who upvoted this, and why?

I agree, I've no idea why it would be at the top of HN. Maybe China-hate?

Hate is a strong word. Dismissing it as simply 'hate' is emotive language (and dishonest, I think) in that it tries to guide the reader to the conclusion that it is completely unreasonable. A more honest word to use would be 'mistrust'

"Hate" is a word that has a variety of meanings, and I was using it as a suffix - it's basically a synonym for '-aggro', a way to describing a pattern of oppositional or antagonistic behavior. (My usage is colloquial. If I were saying what you have mistakenly read it to mean, I would have mentioned hatred of China", or maybe "China hatred".)

I am not using it "dishonestly", and I think that the conclusion it guides people to is completely reasonable and accurate - we should dismiss the majority of opinions that take the form "yup, that's just like China" or "China is doing this shit again? what a surprise."

The behaviors of individual workers in a system where they're being incentivized improperly don't have anything to do with China's current geo-political actions, and I get tired of so many people acting like 'China' (or 'the GOP' or 'California') is a single coherent entity with self-consistent behavior at all scales.

You're right, I read the phrase a little too literally. Apologies!

It was good clickbait. The title made me think something similar to the recent security issue with the University of Minnesota[1] was happening again, but from Huawei this time

[1] - https://news.ycombinator.com/item?id=26887670

The point is that if you were serious about fixing the spelling stuff, you'd put it into a single patch and not break it apart.

The fact that all these fixes are in separate patches means that its being used to game a Key Performance Indicator.

There is an old story of two surgeons - one with a near zero patient death rate and one with a much higher patient death rate. But the surgeon with the higher death rate is tackling harder more complex surgery on sicker patients - while the other avoids the complex stuff.

There is nothing wrong with either.

Similar thing happened with the Hacktoberfest incident last year: https://blog.domenic.me/hacktoberfest/

In short, DigitalOcean unwisely incentivised a bunch of people to make PRs on open source repositories unaffiliated with DO so they could get a T-Shirt. It went so far that some guy actually made a very popular YT video about how you can make a PR for a small typo fix (I guess I still fail to understand why you'd need a YT video to explain that). Maintainers were none too pleased.

Without any context it looks like someone overreacted. It's essentially saying anybody else than Huawei sending cleanup patches is welcome but Huawei is not. Then they try to backtrack that statement by putting out a long list of things which aren't comparable in complexity to the original topic.

Without more context I bet this thread is going to go off-topic.

Sounds like he is saying. Huawei is doing a lot of trivial cleanups to be able to say like they are a major contributor. Basically the argument is:

- trivial patches are ok, if you are just doing it for practice. Like a student.

- trivial patches are not ok, if you are doing a lot of them and only for the purpose to get a higher contribution ranking (like commits per company)

For me as well. This discussion plus recent 'kernel should use github or similar" articles recently makes me think about sponsors.

Doesn't Kernel use sponsors approach, like Debian used to for packages? You can push anything, but you find yourself a competent maintainer who filters your work. Like puts your trivial commits in bulk ;)

I don't think Qu says that Huawei isn't a major contributor.

See:

> > My contributions to the kernel in the past have mainly been on optimizing the performance of the ARM64 SMMU driver, > > including the iova optimization, strict mode optimization, and the lazy mode optimization. Also working on the > > development of some ARM SoC drivers.

> You indeed have done solid contribution to the kernel in the past, thus > better could have been done.

Also:

> Even without checking the git log, I can easily think of some big > contributions from your employer, like EROFS and F2FS. > Thus I don't have any doubt about that.

The remark close to the end, that this is just further damages Huawei's already damaged reputation, especially after beginning with "I did the same at some point", hints at a personal issue rather than professional objection.

There's no professional reason to take such an unrelated jab so it feels more like this individual just found an opportunity to settle a score, or has a different problem with Huawei that's harder to support publicly.

Yep. This is incredibly petty and unprofessional from the reviewer. Keep the ridiculous geopolitical slapfights out of the Linux kernel.

I think the missing context is that Huawei is one of the largest technology companies in the world, every thing they do is built on linux, improving it would improve their business but their contribution to the project is a spell checker. Ok but clearly not their finest work.

NSA has implemented backdoors in software through bug fixes that appear to be just benign typo or spelling fixes but actually allow an exploit in combination with some other exploit etc. To any one accepting the patch it just looks like a typo fix maybe a little odd but nothing dangerous.

Huawei have the people with the skills to do really impressive work but they spend that time and resources fixing typos, this is a little odd.

[Citation needed]

>NSA has implemented backdoors in software through bug fixes that appear to be just benign typo or spelling fixes but actually allow an exploit in combination with some other exploit etc.

I dunno why you needed a citation for this it is literally the NSA's job to do this, if you weren't just trolling me you must be living under a rock.

https://www.wired.com/2014/04/nsa-exploited-heartbleed-two-y...

The context is lack of consideration for people's time.

To use an analogy, imagine you wrote a proposal at work and asked for feedback from your boss. Instead of a single substantive response, they send back a few dozen individual emails, each one a comment on word choice, or font size, or a suggestion for a paragraph break. Even if each one of them might have some minor merit, it was done in the most time wasting way possible, maybe because their boss measures their work output by how many emails they send.

Huawei isn't the kernel boss, but they are essentially submitting feedback. Qu is saying that, for a company the size of Huawei, one that is massively reliant on Linux and has developers' time dedicated to it, he expects them to make their own submissions in a less time wasting fashion. He also suggests the proper way to do it.

https://youtu.be/fMeH7wqOwXA

> If you're hired to do this stuff, you're on your own, you better know what you're doing.

Huawei pays professional developers to contribute to the kernel. Of course they should be held to a higher standard. They should be working on something substantial, not fixing minor problems. Surely there are far more important things to work on. Failing to prioritize important issues coupled with incentives for kernel contribution means they are putting in minimum effort for maximum personal gain at the expense of maintainers.

can you quote the part that supports

>>It's essentially saying anybody else than Huawei sending cleanup patches is welcome but Huawei is not.

? I don't think you can because the article does not say that.

The tone of the email reminds me a lot of Linus Torvald’s brutal code review comments in the google group back then.

Can it be said that the “no nonsense” behavior often exhibited by maintainers of Linux kernel could be one of the reason the project grew to become what it is today

Sure.

But that could be good or bad depending on whether you think Linux has lived up to its potential or not.

Google group? Do you mean LKML? As the name says, it's a mailing list.

Since there are talks about padding numbers for KPI, I did a quick analysis of the linux kernel for single line changes that belong to commits with only one file change and this is what I got for the last 90 days:

https://public-001.gitsense.com/insights/github/repos?q=comm...

Scroll down to the bottom for a breakdown of the file types.

To review the file changes (click on the files) and commits, look at the following:

https://public-001.gitsense.com/insights/github/repos?p=comm...

Disclaimer: I'm the creator of GitSense and there is a bug where the window size won't show 90, but it is for 90 days.

> already broken reputation.

What's that "broken reputation"? Apart from the media circus about 5G. Anything that Huawei has done wrong related to the Linux kernel or the open source community generally speaking?

Metric brushing, clearly, not the nationalistic panic over 5G and spying concerns.

The maintainer says there has been a torrent of trivial fixes from Huawei, and this is just more of the same. Someone wants to get ahead at work on the back of Linux maintainers, just like in the recent example with the University of Minnesota.

You can look through their wikipedia page and see that a lot of early success for the company was based on stolen information from companies like Cisco.

https://securityboulevard.com/2020/12/was-this-huaweis-faile...

Well, nice to see this kind of bullshit isn't just happening to small Open Source projects. Unfortunately, it seems like these kinds of commits are the vast majority of the commits I get on my projects.

It's unfortunate since ideally any contribution is a good contribution, but the Linux kernel has been so highly dependent on maintainers, that them being highly protective of their time is totally warranted.

As someone who works at a big corp with regular OSS contributions, this doesn't surprise me at all. Recently a "Look at how amazing we are" e-mail was sent out to my team touting the number of PRs submitted to a widely-used OSS project.

If your management incentivizes it, you'll try to game the system.

Happens in CPython as well, I like the term "KPI grabbing". If you complain, the central committee will denounce you as non-inclusive.

Nice that there is still some sanity on LKML.

LLVM has quite a number of people who have "review after commit" privilege. These are people who are trusted to know the difference between changes needing review and those that don't. As a result, people can fix typos, fix formatting, etc. without burdening reviewers. Those sorts of changes don't go through the usual review process, but on occasion may be reverted, by either the committer or someone else who has commit privilege.

Remember when they tried to push a backdoor into the kernel last year? I am surprised with how they kept allowing them to submit patches. They acted much more boldly with the issue with the university of Minnesota.

I am not a grsec fan but here https://grsecurity.net/huawei_hksp_introduces_trivially_expl...

Qu Wenruo is not a Linux kernel maintainer. The submission title is incorrect and over dramatic.

Aren't these sorts of trivial fixes best sent to the kernel janitors mailing list?

https://lore.kernel.org/kernel-janitors/

> It's OK for first-time/student developers to submit such patches

That's not the same as "anyone but huawei" as the parent comment noted.

Huawei is not being singled out here because huawei or china but because of a specific behavior that's being called out here.

It looks like fundamental problem is that contributors are rated based on number of commits vs. qulaity of their contirbutions. They should fix that instead of discouraging contributiors.

This same sort of thing is killing stackoverflow haha.

Looks like LKML purged all mail of June 18?

The thread is on the left, up to Jun 22.

Can someone explain this line to me please

    what you guys are doing is really KPI grabbing.

There's a movement away from evaluating people or teams qualitatively, which is difficult and prone to dumb biases, and towards quantitative measures, which are easy and prone to dumber biases.

Presumably here (I don't know if it's been proven) people in that org are being measured by how many commits they make, ignoring the depth or quality of the commits. Because it's someone's idea of a Key Performance Indicator (KPI.) So people are "grabbing" easy commits.

In case anyone forgot about this: https://grsecurity.net/huawei_hksp_introduces_trivially_expl...

Now I understand the story: Qu was salty because his debug message was deleted by the person from Huawei. Qu was mad. Such a trivial event, and still made to HN because of a few keywords.

This is going to become a more common theme. Geopolitics spill into everyday life in all sorts of unexpected ways.

This issue doesn't have anything to do with geopolitics.

I believe that it does on a second order condition. Huawei is under tighter scrutiny because of current events in geopolitics. Were it Google or MIT, I believe it reasonable to expect that the maintainers wouldn't be as prescriptive.

Hi Leizhen, and guys in the mail list,

Recently I find one patch removing a debug OOM error message from btrfs selftest.

It's nothing special, some small cleanup work from some kernel newbie.

But the mail address makes me cautious, "@huawei.com".

The last time we got some similar patches from the same company, doing something harmless "cleanup". But those "fixes" are also useless.

This makes me wonder, what is really going on here.

After some quick search, more and more oom error message "cleanup" patches just show up, even some misspell fixes.

It's OK for first-time/student developers to submit such patches, and I really hope such patches would make them become a long term contributor.

In fact, I started my kernel contribution exactly by doing such "cleanups".

But what you guys are doing is really KPI grabbing, I have already see several maintainers arguing with you on such "cleanups", and you're always defending yourself to try to get those patches merged.

You're sending the patch representing your company, by doing this you're really just damaging the already broken reputation.

Please stop this KPI grabbing behavior, and do real contribution to fix the damaged reputation.

^^ Original message

Not sure where he said it'd be okay if anyone else sent it - but I can see why a string of "cleanup" from a company can be seen as more as metric manipulation than any actual worthwhile contribution.

Can you please not copy the entire OP like that? It's confusing, obscures your specific point, and leads other commenters to reply to random things in the OP, which is fine of course, but that's what the thread is for in the first place.

We detached this comment from https://news.ycombinator.com/item?id=27630257.

Well the reply was to a comment that didn't seem to have read the link so I felt it was apt but alright.

Second-stringers do this. They can accumulate all of the certifications, bogus "patches," and patents they want, but they'll still have zero street-cred. What's more, companies who promote and advance based on these factors are only fooling and embarrassing themselves with weak staff.

The other problem is a fair number of Eastern managers don't know or care about the business or management details, they all of want the prestige of a title with the least effort. A lot of no-name businesses in China are more BS than corporate ones elsewhere. After-all, why is there such a huge market for white business actors?

One KPI in which Linux dominates: drama.

> "you're really just damaging the already broken reputation."

Actually, with this email and the viral on HN, I think this quotation is true for Qu Wenruo. He/she has just damaged the already broken reputation.

Not only Huawei, but also other companies will take a look at this accident to get the learned lessons.

Oh no! Won't somebody please think of the companies?!

There is much hostility against Huawei. In the email, the message title was really offensive and even assumed something about KPI, as if the Huawei engineer was inventing patches in order to count as more contributions.

Even the editorialized subject of the post on HN is polemic.

What is this problem with Huawei? It is the US government that is doing the economic war against China and because of that, decided to cripple Huawei.

Reply on news.ycombinator.com