Learnings from kCTF VRP's 42 Linux kernel exploits submissions
security.googleblog.com
security.googleblog.com
It was submitted under graplsecurity.com, chompie has since mirrored the research as the company no longer exists.
The title makes it look like a design problem with io_uring, Google's reaction makes it feel like a design issue. But the article itself sounds like growing pains.
Very generally speaking, not really. It's a huge class and the pace of discovery of new vectors has continued nearly unabated since Spectre. See https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner... The effectiveness and cost of various remediations--software, microcode, hardware--varies depending on the particular vector, so some vectors can be considered fixed, while other vectors, such as some of those relating to SMT, might never be fully fixed without effectively giving up some of the benefits of SMT.
Some of the more robust theoretical remedies, like cache coloring and kernel scheduler-based process affinity schemes, still haven't seen much investment or attention, so I would expect the state of things to remain relatively crappy. It just may seem less crappy because the industry has become inured to the threat, and also perhaps because there's usually much lower hanging fruit for attackers (i.e. traditional exploits like with io_uring), so these vectors are rarely used in the wild.
Weird, I was told Zen3 is fixed.
I heard from Zen3 silicon, spectre and friends are fixed.
In this case the only thing one can fix is, surprise, the vulnerable code!
1. Security has been sort of an afterthought. io_uring was being built out as a complex, powerful capability and only much later were things like auditing, selinux hooks, and ebpf even considered.
2. io_uring is inherently complex code. It's highly optimized, concurrent, and new. It does a lot of work to avoid things like copying across the kernel boundary. This code is hard to get right under the best of circumstances.
3. io_uring is very different. This is basically a new system call interface that behaves in a different way from the other interface, we're playing catch up in security.
So in theory this could all be fixed but it's going to take a lot of work and, importantly, a shift in priorities - io_uring can't just be developed as the "gotta go fast" interface, it needs to be designed around safety.
> While io_uring brings performance benefits, and promptly reacts to security
> issues with comprehensive security fixes (like backporting the 5.15 version
> to the 5.10 stable tree), it is a fairly new part of the kernel. As such,
> io_uring continues to be actively developed, but it is still affected by
> severe vulnerabilities and also provides strong exploitation primitives. For
> these reasons, we currently consider it safe only for use by trusted
> components.
Ensuring that io_uring is covered by the Linux Security Modules (LSM) framework, that provides hooks for things like AppArmor or SELinux (and a variety of others, some even can even get stacked), was only done after it was already quite popular.
So FWICT, even if io_uring is existing since a while, 1) the security experts vetting it are looking only closer since relatively recently and 2) it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel, especially as the project still wants to stay high performance.
IMO, this doesn't mean io_uring is inherently unsafe, or has no use, but that one certainly needs to evaluate if great async submission model is worth the potential security implications. E.g., in an isolate environment, where no untrusted third-party apps or people have access, it can be fine, while especially in environments that Google runs it isn't worth the trouble for now.
But, I think that the features which io_uring brings are too nice to have for quite some (I/O or syscall heavy) workloads even in those environments, so there will be an incentive it to the integration into the existing Linux security stack and mechanisms, and evolve the latter to be also a good fit for this rather different interface.
It's less about integration into the security stack. It's just that the implementation of io_uring is hard to get right, and at 2023 there seems to be an endless stream of race condition bugs. Quite like user namespaces ten years ago. It doesn't matter how good your security design is. Hackers just find a memory corruption bug and get around everything.
To be fair it isn't endless, finding an io_uring bug which can be exploited to cause memory corruption and ultimately kernel code execution in 5.15 tree is measurably harder now. But io_uring moves very fast and it's an entirely different story for mainline kernels.
> It's less about integration into the security stack.
>> it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel
> Hackers just find a memory corruption bug and get around everything.
Because I think once you get down to the kind of code being written here it's going to be chock full of unsafe {} and UnsafeCell and various snakey error-prone tricks to make the borrow checker work in the context of the entirely different world that is the kernel, closer to the metal. And so realistically most of the same issues that impact C code will happen here.
I haven't looked at the io_uring security issues in general but I've used io_uring (from Rust) and I can see how it could go wrong. And I can see that if one was writing a facility like this in Rust it could end up subject to many of the same issues, because in the end io_uring is holding references to user instructions and executing them in a pretty privileged context. Many things could go wrong here: how one manages ownership, what the boundaries between this thing and the rest of the kernel look like, etc. There's no magic bullet that Rust provides there, other than perhaps an expectation of a certain kind of borrowing discipline?
Hopefully Google shutting this down in their GKE environment doesn't spread beyond into AWS and Azure hosted instances, etc. because I think there's plenty of people who will be impacted quite negatively. io_uring has incredible promise, and the kinds of companies that it will hurt are the fairly innovative ones pushing the envelope for database or server tech generally (including a former employer of mine!).
the normal lsm hooks all do something well scoped, specific, and take meaningful parameters.
the io_uring hooks take what appear to be a turing complete language snippets as a parameter.
the hooks now all return false (disabled).
Given how large they are as a target, we'd also hear about them being exploited all the time if that was true.
* ChromeOS, COS and Android are opensource, and they are all up to date. Mind elaborating?
The mere fact that the source is available doesn't mean that someone will take the (potentially unbounded) time to port it to a future release and test it sufficiently.
"As such, io_uring continues to be actively developed, but it is still affected by severe vulnerabilities and also provides strong exploitation primitives."
This is ridiculous. No, they aren't. No one is. Kernels on production Linux devices[1] are cut from an LTS branch and get prompt and expertly audited security backports of everything anyone knows about. Now, sure, that process does break down occasionally; but this is an area being shepherded by some of the best engineering effort in the world. In fact I'd go so far as to say a regularly updated LTS kernel is probably the software product that is most responsive to security issues.
[1] All the ones I know about anyway. Surely there are some gadgets being less prudent, but Android and ChromeOS are absolutely not on the list.
Which lead me down this rabbit hole - https://english.stackexchange.com/questions/546959/why-is-le...
Observation bias, but I first noticed this word popping up in tech talks from Microsoft first.
So an award roughly equal to four months' salary for an average Silicon Valley engineer is considered "heavily investing" in security, when it comes to potentially industry-shattering exploits?
I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.
You also seem to discount the monetary value of not becoming a criminal by selling exploits unlawfully. Do you want to be the person that reported a vulnerability to increase security in the world or to give more power to spy agencies or crime syndicates?
When you sell this to another party, are you going to report it on your taxes? If you do report it, is your buyer going to like that?
If you clear all those conditions, then by all means.
You don't need to declare who gave you the money on the income tax filing.
Don't you? I thought that in the US you needed to have either a 1099 or a W2 for any significant employment income.
- you want the payment sent to your LLC or similar, not directly to you. See above
- the requirement to file a 1099 is on the payer, not the worker
- many transactions don't require filing a 1099, most notably all payments made using a credit card. If you're an independent contractor in the US and accept credit cards, you'll get paid 5 minutes after sending an email instead of 30 days later after it goes through layers of approval
That’s kind of begging the question. Google invests a non trivial amount employing Linux kernel developers. How do these bounty payouts compare in terms of the amount they pay for their own engineers and security researchers? This also isn’t a donation. This is rewards for work Google finds valuable enough that they’ve changed several products based on this work. They also benefit immensely from all the free labor other companies and volunteers out into the kernel.
I think OP raised an extremely fair critique of the amount being too small.
I believe the world governments should create an international organization funded by per country quotas like the IMF for example to pay for bounties, but until then, the fact a private company does this sets the market price for exploits, and saying "it's too low" seems meaningless if there's no proof anyone is willing to pay more.
I'm sure if it were $1m per bounty some other HN user would say it should be more as well. That's why you define this by how much someone _actually_ pays out, because talk is cheap.
The black market is a good proxy for estimating the true market value of an exploit. There’s a coupon to be applied in terms of the legal costs associated with selling on the black market, but still comparable.
The other way to look at it is when are there diminishing returns. From the chart they posted, they’re very far away from that.
Finally the other way to look at it is compared with other kinds of bounties they run for OSS.
Anyway, of course they should be commended for having any bounty. They’re certainly not required by legislation. But maybe instead of having an international governmental or NGO with funding, you could require that some percentage of a SW company’s revenue is diverted to bug bounties for SW depending on how critical the SW is to the ISV’s business. That way it’s a cost directly borne by the ISV using the software to internalize the security risk into a P&L line item for the software being used (and not just OSS - all software from an ISV should be forced into bug bounties like that). And the vendor has a vested interest in confirming the VULN whereas a NGO could have all kinds of soft corruption to incentive flowing payments to friends. That being said, I’m not sure the status quo would really improve in such any alternate scheme.
No. I have no idea. But it would clearly be in the interest of such institutions to have such programs, so common sense says they do. I'm sure that people who are active in this business would know more about it.
Unless you’re a very known quantity they’re not going to talk to you directly, anyways. You’re going to go through a broker who will take a cut. My understanding (though I don’t know much about it) is that say, the NSA, has their own internal teams and they don’t need to buy anything anyways. The other federal agencies will contract out to a handful of firms that generally have salaried employees. You can work there, but now you’re seeing very little of the “profits” that come from the sale. Maybe your exploit will get some shoddy forensics glued into it and end up being sold as “data recovery” or something for a local police department, who probably doesn’t have much money to spare anyway.
If you try to sell your exploit on the black market for 133k but mess up and walk into a federal agent honey pot, now your in jail for 10 years.
Even if you did successfully do it, imagine how hard it is to explain where an extra 133k came from in your bank account.
Imagine though, that this information you willing sold to a shady party was used in a big hack. Say national security or an oil pipeline or something.
Do you think your defense will hold up in court? How many years will you be in court? Even if you are deemed not guilty, will your life ever be the same?
You’re gonna risk all of that because you thought you could squeeze a $200k out of your bug?
Of course not. You start a "security consulting company" and then apply to be a contractor. There are hundreds, if not thousands, of companies like that. And as an individual researcher, you can either join such a company, or freelance for one.
It's indeed not trivial, but I imagine that anyone with enough experience in the business would know how to pull it off.
In fact, despite so many of us personally relying on this software and giving nothing, we are all immune from criticism.
I suspect that if I had offered $10 into a bounty I would be responsible too for not paying more than lunch in SF. Haha!
Common sense says that with bounties like that, they won't attract the most serious exploits, which isn't an ethical problem, but can absolutely be a security problem.
There was a crypto company running with all spectre:meltdown mitigation off. And no one took the money until the company went bust and an insider tried to run with it.
I also told the world that it was the case so it wasn't secret: https://news.ycombinator.com/item?id=32077583
That was a lot of money untraceable if so many of these were exploitable. They're generally not. Google is often the best buyer.
Funny how when a person/company does a small good deed, they get criticized for not doing more but when they don't do anything and implicitly offer $0 they don't get criticized.
If that were the case, why did even a single person report a vulnerability to Google? Why didn't these muppets just sell it to somebody else for ten times as much? I would suggest tha it's because what you claim isn't correct. It's not actually "easy", and the payoffs won't be "a lot more".
Now, it's pretty hard for anyone to actually refute your argument since literally the only evidence you're providing is your imagination. The one thing we do know is that tens of serious vulnerabilities were reported to Google via this program. Those are a concrete existence proof of this VRP being a sufficient incentive for bugs to be reported via this legit channel. Surely that outweighs your complete lack of evidence.
This is unfortunately false. The running price for Linux kernel exploit isn't that high.