The AMD Branch (Mis)Predictor Part 2: Where No CPU Has Gone Before
grsecurity.net
grsecurity.net
This goes to show you that the software and digital logic design space is very different, and you should not assume any amount of experience and knowledge in one will prepare you for the realities of the other.
Sure it seems astounding that a CPU would speculate not-taken over a static, direct, unconditional branch. Surely no sane logic designer would ever do such a thing!
When you realize doing that might require more logic in a downstream stage after fetch and after the branch was decoded, and from there you need more control logic and wires going to other stages and logic to re-steer the pipeline in possibly a different manner or stage than would otherwise be required, and all that requires validation and verification (which can easily become a multiplicative effort), and if doing all of that did not bring you performance gains that outweighed the cost, and if you didn't understand the possible security implication or considered it to also not be worth the cost, then actually it's quite understandable for a reasonable implementation to behave this way.
Even the second quote smells a bit like dunning kruger. Everything has bugs or misfeatures, especially with hindsight. Everyone (including Linus and many other kernel developers) thought that the massive speculation capability and correspondingly astounding performance results on privilege domain crossing for Intel CPUs was the duck's nuts back before spectre/meltdown. And they certainly didn't foresee or discover the security issues with doing that.
(EDIT Disclaimer: Am linux developer and armchair microarchitect)
Also, it’s not like software designers don’t waste cycles. In many languages, for example, arguments to log statements are evaluated even if logging is disabled. There sometimes is the hope that the runtime will figure that out and move that inside the if logging is enabled check, but that’s only a hope (COBOL had the ALTER statement for efficiently handling such things, but that’s being frowned upon :-))
Also: a conditional branch can be always taken, for example when the software does an if cpu supports instruction extension at startup, sets a flag accordingly, and later branches depending on that flag.
That, too, is a case where the CPU will speculatively execute what, to it, may look like garbage. So, you have to make sure speculative execution handles that.
Security, sure arguably it could have been caught there. But Linux kernel development does not have the high ground when it comes to security problems.
Also some amount of speculation even for untaken branches is unavoidable: the fetch stage need to know whether to fetch the next instruction or not before the unconditional jump is decoded, which is multiple pipeline stages downstream.
Maybe upstream is able to work with hundreds of different and competing companies and thousands of different people and clashing personalities and opinions, but they just happen to irrationally refuse to treat grsecurity fairly.
Or maybe it's the simpler explanation that has more obvious motives.
But those types of patches were precisely what grsecurity was pushing. So you basically had a fundamental clash of philosophies (independent of the clash of personalities) about whether the work grsecurity was doing was even valid from a software engineering standpoint. Arguably because of the relentless onslaught of exploits and growing enterprise influence, the Linux community slowly relented, but never with a mea culpa, AFAIK.
A similar dynamic played out with /dev/urandom, where the Linux community and subsystem maintainers were adamant that the existing blocking and entropy accounting behaviors were crucial to security and non-negotiable. But now look where things stand--both are gone after 10-15+ years of debate. Again, without any mea culpas.
Ditto for sandbox-friendly interfaces. Once upon a time any grievance that an interface which relied on /proc, /dev, or similar was suboptimal from a sandboxing standpoint (kernel surface area, headaches with chroots, etc) was met with derision. Eventually we got getrandom(2), and other recent application interfaces, like process descriptors, were revised so they were usable without /proc or similar.
You could see alot of this play out on Slashdot, HN, etc as commenters frequently parroted Linux developers' talking points and assertions.
There's plenty of blame to go around.
Sure Linus has been hostile or funny about such things at times, but that has not prevented him from being convinced or them from being merged.
> But Grsecurity made their patches freely available for the vast majority of their existence.
I didn't say anything about whether they made their patches freely available or not. Not sure what that has to do with what I wrote.
> I think it just shows ignorance of the situation to imply that they're only interested in making money.
I didn't imply that. I said they have a vested interest in perception of being more secure than upstream. Which they do. Replying with vague snark and going off on some wild tangent imagining things that I never implied isn't helpful if you can't address what I wrote.
I didn't say anyone is to blame as such, nor does grsecurity have any requirement or moral duty to put effort into upstreaming their work. But if you look at the motivation there is a pretty reasonable explanation why they have not done so, in my opinion that's more reasonable and likely than the idea that it's upstream being particularly unfair or uncooperative to this one group.
3: https://www.spinics.net/lists/kernel/msg2540934.html
2008 (Linus):
>>I think the OpenBSD crowd is a bunch of [self-stimulating] monkeys
2019 (Greg KH):
>>OpenBSD was right on disabling hyper-threading
Ahh that's why OPNsense uses it? But hey you write like a real professional in that field so i believe you, instead the people who actually work day in day out on a professional firewall product.
Maybe you have to acknowledge that there is just one clown here ;)
It's worth noting that the memory protection scheme, introduced with the 286, was never intended to be a strong security barrier, but instead a means of isolating bugs and making them easier to debug.
Not always. When using AES algorithm the key gets expanded and the expanded form can be distinguished from random data. Combination of RSA private key and public parameters should also be distinguishable from random data due to integer multiplication properties RSA uses. There are tools for searching encryption keys in program memory using such tricks to recognize them. Such tools would typically be used when a program like game or DRM media player runs on attacker controlled computer, but in theory would also help when fragments of memory can be leaked from a server due to side channel attacks or bugs like heart bleed.
I do worry sometimes that something is up with CPU development, that we're tending towards more and more complicated designs with workflows that are very hard to analyse and simulate even for the designers themselves, but the actual workload execution ability performance per core isn't shifting upwards all that much, and then weird mitigations have to be applied that reduce that execution ability in practice.
Something makes me think that perhaps a different design paradigm should prevail, with particular attention paid to segregation of workloads and of core partitioning, perhaps an abandoning of hyperthreading and even to the extent of having 100% physical separation of cores and their caches.
But I'm very much not an expert in the field.
A little birdie inside of me every now and then wakes up and whispers 'is it a coincidence that these design paradigms are yielding so many vulnerabilities?'
It's a bit concerning that we are not really making amazing progress in single threaded performance.
I think we might not be going quite far enough with complex instructions, and maybe we need to be able to define out own with custom microcode or something.
Like, a CPU that had just a few instructions, enough to bootstrap and that's it, and a very fast FPGA that could switch configuration quickly to make your own instructions.
Right now it takes time to send stuff to the GPU. We need acceleration we can access in one instruction, so that normal compilers can generate it all behind the scenes.
Or, alternately, modular pluggable acceleration. You could fit like 8 little MicroSD sized special purpose accelerators on a laptop MB.
Maybe file serving can be hardware accelerated. Is there an actual reason you need a beefy server to serve a ton of media, or can CDN caching be optimized the way ethernet switches are, so you have a tiny credit card size device serving 10gbps on three watts?
Encryption aside a lot of file serving is accelerated. In Linux, apache has just called a kernel function (I forget what it's called, something unsurprisingly along the lines of socket_sendfile) to get data out on the wire, and that function itself is hardware accelerated by the network hardware through the large send offload v1 & v2 functions offered by the network card driver.
(also not an expert however)
Not to mention that the whole idea of the Mill requires a “sufficiently smart compiler”. We tried VLIW (based on the same compiler ideals) with the Itanium and it was a major flop.
The Itanium platform was totally unaffected by Spectre/Meltdown type attacks, you may note.
Intel invested heavily into Itanium, and the 90s started seeing the deaths of competing processor architectures in part because of how hyped up the Intel Itanium was getting. It was intended to be the 64-bit version of x86, hence the abbreviation IA-64. AMD was the one who created what we now know as x86-64, and my understanding is that Microsoft more or less forced Intel to implement AMD's x86-64 specification.
In its later years (let's say by 2010, since that's the midpoint of Itanium's life as a shipping product, but I don't have any clear dates as to when the shift happens), it does seem to be that Intel was reluctant to continue supporting Itanium. But that was definitely not the case beforehand.
Ultimately, the compiler never materialized and I'm convinced they knew it wouldn't (but in the meantime, they almost killed all the RISC competition). The Halting problem means that at best you have heuristic optimizations that constantly fall through.
The only way to solve these problems with a high degree of success is to analyze the code as it is running which is what other CPUs actually do. This is so true that later versions of Itanium (latest being released in 2017) were just VLIW wrappers around a rather traditional, speculative core.
You are correct in as much that achieving this level of optimization through static analysis alone without cooperation in the language would be a very challenging problem. The folks at https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/ are taking a stab at it nonetheless - their approach requires language extensions.
I'd prefer not to ignore the issues of Spectre and Meltdown - doing so is what got us here in the first place. I still run many older machines in roles for which they are well suited and would gladly trade performance for more predictable and secure behavior.
Intel didn't want to make 64-bit x86, but the market prefered amd64 over itanium.
Maybe their BTB is that good that they didn't see it worth investing in the control logic for it.
Pick two:
Shared Physical Hardware
Performance
Security
Not pleasantly, the FH fiction wasn't, either.
IIRC the term was introduced with the Return Oriented Programming style of attacks and reused for spectre-like speculation attacks.
Smart people will look at the current executable code in the system, and work out how to manipulate the existing executable code to build the attacker's programs (return oriented programming); the identified chunks are called gadgets.
As an aside, given how Intel squandered its near-monopoly for years until AMD came up from behind and kicked 'em in the pants, I'll be sticking with AMD out of principle. I'm especially upset at some of the really dirty tricks Intel pulled in the past.
Random IOPS is off the charts faster on Intel my 1:1 tests between real-world system with the same drive. I had mediocre (but suitable) experiences with Athlon chips 22 years ago, with the same basic outcome: great CPUs, mediocre platform (chipset engineering etc).
I understand your emotions about AMD and Intel, you may not be "wrong". I have no strong feelings on that, just want what works. But for holistic engineering my 40 years of PC building experience is persuasive towards Intel products.
Indeed. Especially for "AAA" games designed to run on major AMD-based consoles such as Xbox and Playstation. Those games might actually be slightly more likely to be broken on Intel systems!