Even worse: They bragged about it, then sent a new wave of buggy patches to see if the "test subjects" fall for it once again, and then tried to push the blame on the kernel maintainers for being "intimidating to newbies".
This is thinly veiled and potentially dangerous bullying.
Fortunately, the episode also suggests that the kernel-development immune-system is fully-operational.
"A lot of these have already reached the stable trees. I can send you revert patches for stable by the end of today (if your scripts have not already done it)."
https://lore.kernel.org/linux-nfs/YH%2F8jcoC1ffuksrf@kroah.c...
It looks like actual security vulnerabilities were successfully added to the stable branch based on that comment.
It had a high human component because it was humans making decisions in this process. In particular, there was the potential to cause maintainers personal embarrassment or professional censure by letting through a bugged patch. If the researchers even considered this possibility, I doubt the IRB would have approved this experimental protocol if laid out in those terms.
Not to excuse them at all, I think the results are entirely appropriate. What they're seeing is the immune system doing its job. Going easy on them just because they're a university would skew the results of the research, and we wouldn't want that.
The only relevant question is: "Will the investigator use ... information ... obtained through ... manipulations of those individuals or their environment for research purposes?"
which could be idly thought of as "I'm just sending an email, what's wrong with that? That's not manipulating their environment".
But I feel they're wrong.
https://grants.nih.gov/policy/humansubjects/hs-decision.htm would seem to agree that it's non-exempt (i.e. potentially problematic) human research if "there will be an interaction with subjects for the collection of ... data (including ... observation of behaviour)" and there's not a well-worn path (survey/public observation only/academic setting/subject agrees to study) with additional criteria.
Besides, all we have to do is look at the outcome: Outrage on the part of the organization targeted, and a ban by that organization that will limit the researcher's institution from conducting certain types of research.
If this human-level harm was the actual outcome means the experiment was a de fact experiment including human subjects.
One of the important rules you must agree to is that you cannot deceive anyone in any way, no matter how small, if you are going to claim that you are doing exempt research.
These researchers violated the rules of their IRB. Someone should contact their IRB and tell them.
If the IRB approved this as exempt and they had an accurate understanding of the experiment, it makes me question the IRB itself. Whether the researchers were dishonest with the IRB or the IRB approved this as exempt, it's outrageous.
I do recommend participating more in other threads and a little less in this thread, where you're repeating pretty much the same point over and over.
[1] https://twitter.com/SarahJamieLewis/status/13848713855379087...
To be clear, I don't think what the kernel maintainers did is wrong; it's just sad that all past and future potentially genuine contributions to the kernel from the university have been caught in the crossfire.
I'm not sure how the law works in such cases, but surely the IRB would eventually have to realize that an explicit denouncement by the victims means that the "research" cannot go ahead
Eg - If you want to do kernel related research, don’t go to the university of Minnesota.
- Human subjects - Intentionally misleading/misrepresenting things, potential for a lot of damage, given how widespread Linux is - No informed consent at all!
Sorry but one cannot use unsuspecting people as guinea pigs for research, even if it is someone from a reputable institution.
Sure they can only automatically ban the .edu address, but it would be pretty meaningless to just ban the university email host, but be ok with the same people submitting patches from personal accounts.
I would also explicitly ban every person involved with this "research" and add their names to a hypothetical ban list.
As Joe Blow at home who happens to go to school or work there you could submit even if you were part of the research team. Because you are not representing the university. The university is banned.
The reputational damage will be lasting, both for the researchers, and for UMN.
Which itself could be the basis of a follow up research paper. The first one was about surreptitiously slipping vulnerabilities into the kernel code.
There's nothing surreptitious about their current behavior. They're now known bad actors attempting to get patches approved. First nonchalantly, and after getting called out and rejected they framed it as an attempt at bullying by the maintainers.
If patches end up getting approved, everything about the situation is ripe for another paper. The initial rejection, attempting to frame it as bullying by the maintainers (which ironically, is thinly veiled bullying itself), impact of public pressure (which currently seems to be in the maintainers' favor, but the public is fickle and could turn on a dime).
Hell, even if the attempt isn't successful you could probably turn it into another paper anyway. Wouldn't be as splashy, but would still be an interesting meta-analysis of techniques bad actors can use to exploit the human nature of the open source process.
Seems more like low grade journalism to me.
Wouldn't the correct term for that be: malicious threat actor?
Red team penetration testing doesn't involve the element of surprise, and is pre-arranged.
Intentionally wasting peoples time, and then going further to claim you weren't, is a malicious act as it intends to do harm.
I agree though, it's fascinating but only in the true crime sense.
The existing paper researched the feasibility of unknown actors to introduce vulnerable code. The hypothetical second paper has the same basis, but is from the vantage point of a known bad actor.
Reading through the mailing list (as best I can), the maintainer's response to the latest buggy patches seemed pretty civil[1] in general, and even more so considering the prior behavior. And the submitter's response to that (quoted here[2]) went to the extreme end of defensiveness. Instead of addressing or acknowledging anything in the maintainer's message, the submitter:
- Rejected the concerns of the maintainer as "wild accusations bordering on slander"
- Stating their naivety of the kernel code, establishing themselves as a newbie
- Called out the unfriendliness of the maintainers to newbies and non-expects
- Accused the maintainer of having preconceived biases
An empathetic reading of their response is that they really are a newbie trying to be helpful and got defensive after feeling attacked. But a cynical reading of their response is that they're attempting to exploiting high-visibility social issues to pressure or coerce the maintainers into accepting patches from a known bad actor.
The cynical interpretation is as much social-exploit-vector vulnerability research as what they did before. Considering how they deflected on the maintainer's concerns stemming from their prior behavior and immediately pulled a whole bunch of hot-button social issues into the conversation at the same time, the cynical interpretation seems at least plausible.
[1] https://lore.kernel.org/linux-nfs/YH5%2Fi7OvsjSmqADv@kroah.c...
[2] https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
Once any maintainer of the community responds to the email,indicating “looks good”,we immediately point out the introduced bug and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
> If it quacks like a duck and waddles like a duck, then it is a duck.
A lot of horrible things have happened on the Internet by following that philosophy. I think it's imperative to learn the rigorous facts and different interpretations of them, or we will continue to great harm and be easily manipulated.
As soon as I read that all sympathy for this clown was out the window. He knows exactly what he's doing.
I'm not saying people or institutions cant change. But the burden of proof is on them now to show that they did. A good first step would be to acknowledge that there IS a good reason for doubt, and certainly not whine about 'preconceived bias'.
It is not the job of the kernel maintainers to justify the teams new nonsense patches. If the team has stopped being bullshit, they should defend the merit of their own patches. They have failed to do so, and instead tried to deflect with recriminations, and now they are banned.
I don't have data to back this up, but I've been around a while and I can tell you papers are rejected from conferences for ethics violations. My personal observation is that infosec/cybersecurity academia has been steadily moving to higher ethical standards in research. That doesn't mean that all academics follow this trend, but that unethical research is more likely to get your paper rejected from conferences.
Submitting bugs to an open source project is the sort of stunt hackers would have done in 1990 and then presented at a defcon talk.
IEEE seems to have no problem with this paper though.
>>> On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits Qiushi Wu, and Kangjie Lu. To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
That doesn't stop someone from lying about it, but it's not a casual claim, and doing so would probably bring community censure (as well as being easily falsifiable after time).
Edit: All conference are different, I dont know if it applies to that one.
Security research is not always the most ethical branch of computer science, to say it mildly. Those are the people selling exploits to oppressive regimes, allowing companies to sit on "responsibly reported" bugs for years while hand-wringing about "that wasn't in the attacker model, sorry our 'secure whatever' we sold is practically useless". Of course the overall community isn't like that, but the bad apples spoil the bunch. And the aforementioned unethical behaviour even seems widely accepted.
> We send the minor patches to the Linux community through email to seek their feedback. Fortunately, there is a time window between the confirmation of a patch and the merging of the patch. Once a maintainer confirmed our patches, e.g., an email reply indicating “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our correct patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. All the UAF-introducing patches stayed only in the email exchanges, without even becoming a Git commit in Linux branches. Therefore, we ensured that none of our introduced UAF bugs was ever merged into any branch of the Linux kernel, and none of the Linux users would be affected.
It seems that the research in this paper has been done properly.
EDIT: since several comments come to the same point, I paste here an observation.
They answer to these objections as well. Same section:
> Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.
And, coming to ethics:
> The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.
How is wasting the time of maintainers of one of the most popular open source project "done properly"?
Also, someone correct me if I'm wrong, but I think if you do experiments that involve other humans, you need to have their consent _before_ starting the experiment, otherwise you're breaking a bunch of rules around ethics.
> Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.
And, coming to ethics:
> The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.
Not sure how that passage justifies wasting the time of these people working on the kernel. Because the issues they pretend to fix are real issues and once their research is done, they also submit the fixes? What about the patches they submitted (like https://lore.kernel.org/linux-nfs/20210407001658.2208535-1-p...) that didn't make any sense and didn't actually change anything?
> And, coming to ethics:
So it seems that they didn't even just mislead the developers of the kernel, but they also misled the IRB board, as they would never approve it without getting consent from the developers since they are experimenting on humans and that requires consent.
Even in the section you put above, they even confess they need to interact with the developers ("this experiment will take certain time of maintainers in reviewing the patches"), so how can they be IRB-exempt?
The closer you look, the more sour this whole thing smells.
I was wondering why he banned the whole university and not just these particular researchers. I think your quote is the answer to that. I'm not sure on what basis this exemption was granted.
Here's what the NIH says about it:
Definition of Human Subjects Research
https://grants.nih.gov/policy/humansubjects/research.htm
Decision Tool: Am I Doing Human Subjects Research?
https://grants.nih.gov/policy/humansubjects/hs-decision.htm
And even if they did find some way to justify it under their own rules, some of the research subjects clearly disagree.
https://lore.kernel.org/lkml/20210421130105.1226686-8-gregkh... for the full list
Inasmuch as the IRB marked this as "not human research" they appear to have erred.
Even if none of the patches made into the kernel (which doesn't seem to be true, according to other accounts), it's still possible to do permanent damage to the community of kernel maintainers.
I would have reacted the same way Greg did - I don't care what credentials someone has or what their hidden purpose is, if you are intentionally submitting malicious code, I would ban you and shame you.
If particular researchers continue to use methods like this, I think they will find their post-graduate careers limited by the reputation they're already establishing for themselves.
When I fill out the NIH's "is this human research" tool with my understanding of what the study did, it tells me it IS human subjects research, and is not exempt. There was an interaction with humans for the collection of data (observation of behavior), and the subjects haven't prospectively agreed to the intervention, and none of the other very narrow exceptions apply.
Essentially, the researchers were not in control to stop the experiment if it deviated from expectations. They were relying on the exact system they were testing to trigger its halt.
We also don't know what details they gave the IRB. They may have passed through due to IRB's naivete on this: It had a high human component because it was humans making many decisions in this process. In particular, there was the potential to cause maintainers personal embarrassment or professional censure by letting through a bugged patch. If the researchers even considered this possibility, I doubt the IRB would have approved this experimental protocol if laid out in those terms.
We can independently conclude this kind of research has put open source projects in danger by getting vulnerabilities that could carry serious real world consequences. I could imagine many other ways to carrying out this experiment without the consequences it appears to have had, like perhaps inviting developers to a private repository and keeping the patch from going public, or collaborating with maintainers to set up a more controlled experiment without risks.
This seems by all appearances an unilateral and egoistic behavior without great thought into its real world consequences.
Hopefully researchers learn from it and it doesn't discourage future ethical kernel research.
"In the past several years, we devote most of our time to improving the Linux kernel, and we have found and fixed more than one thousand kernel bugs"
But someone upthread posted that this group has a total of about 280 commits in the kernel tree. That doesn't seem like anywhere near enough to fix more than a thousand bugs.
Also, the clarification then says:
"the extensive bug finding and fixing experience also allowed us to observe issues with the patching process and motivated us to improve it"
And the way you do that is to tell the Linux kernel maintainers about the issues you observed and discuss with them ways to fix them. But of course that's not at all what this group did. So no, I don't agree that this research was done "properly". It shouldn't have been done at all the way it was done.
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
IEEE is just the publishing organisation and doesn't review research. That's handled by the program committee that each IEEE conference has. These committees consist of several dozen researchers from various institutions that review each paper submission. A typical paper is reviewed by 2-5 people and the idea is that these reviewers can catch ethical problems. As you may expect, there's wide variance in how well this works.
While problematic research still slips through the cracks, the field as a whole is getting more sensitive to ethical issues. Part of the problem is that we don't yet have well-defined processes and expectations for how to deal with these issues. People often expect IRBs to make a judgement call on ethics but many (if not most) IRBs don't have computer scientists that are able to understand the nuances of a given research projects and are therefore ill-equipped to reason about the implications.
Could a number of seemingly unrelated individuals introduce a number of bugs over time to form and exploit without being detected?
Doing code review at work I am constantly catching blatantly obvious security bugs. Most developers are so happy to get the thing to work, that they don't even consider security. This is in high level languages, with a fairly small team, only internal users, and pretty simple code base. I can't imagine trying to do it for something as high stakes and complicated as the kernel. Not to mention how subtle bugs can be in C. I suspect it is impossible to distinguish incompetence from malice. So aggressively weeding out incompetence, and then forming layers of trust is the only real defense.
I can think of a whole lot of three letter agencies with reasons to do that, most of whom recruit directly from universities.
"Can open-source maintainers make a mistake by accepting faulty commits?"
In addition to being scummy, this research seems utterly pointless to me. Of course mistakes can happen, we are all humans, even the Linux maintainers.
Both are unethical, disruptive, and prove nothing about the integrity of the organizations they target.
Submitting a fake paper to a journal read by a few dozen academics is a threat to someones ego. It is not in the same ballpark as a threat to IT infrastructure everywhere.
https://duckduckgo.com/?q=facebook+emotional+study&t=fpas&ia...
The other part of the patch, that checks for `flow` being NULL may be unnecessary since it looks like the handle is always from an active context. But that's a guess. And it's only unreachable code.
The opinion I have from this is despite other patches being bad ideas, this one doesn't look like it. Because the other patches didn't make it past the mailing list, it demonstrates that the maintainers are doing a good enough job.
Any pentester or red team considers their profession an ethical one.
By the response of the Linux Foundation, this is clearly not authorized nor falling into any bug bounty rules/framework they would offer. Social engineering attacks are often out of bounds for bug bounty - and even for authorized engagements need to follow strict rules and procedures.
Wonder if there are even legal steps that could be taken by Linux foundation.