Intel vulnerabilities costing 25% CPU performance loss to a cloud provider
twitter.com
twitter.com
I'd genuinely be curious to find out what the eventual results are because as i understand it AMD is not too far away from Intel as a standalone processor, surely in a "real world" scenario they'd be significantly faster?
Intel: -13% for the Core i9 7980XE, -17% for the 8086K.
AMD: -3% for the 2700X.
Reference: https://www.phoronix.com/scan.php?page=article&item=linux50-...
Phoronix is due to release new benchmarks tomorrow showing full impact from Spectre/Meltdown/L1TF/MDS. There are some initial benchmarks at https://www.phoronix.com/scan.php?page=news_item&px=MDS-Zomb...
The 9980XE is Skylake. It's not actually a "new" processor at all. Consumer parts were released in 2015 (Core ix-6xxx) & server parts in 2015 as well (Xeon E3-v5).
In fact the 9980XE itself isn't even a new offering in the HEDT space for Intel, as it's basically a rebrand of the 7980XE. The differences are just a soldered heat-spreader instead of paste & a small clock bump to go along with that. It's +200mhz turbo & +400mhz base, complete with a power consumption increase to match.
EDIT: The 9900K (Coffee Lake) does have in-silicon mitigations for Meltdown & L1TF (Foreshadow), though: https://www.anandtech.com/show/13450/intels-new-core-and-xeo...
And while it would be nice to call any point "the end" and measure then, as of two weeks ago they were still finding additional vulnerabilities and patching them. So if there is an end we aren't there yet.
That all being said, I am also curious on what the results would show. Just very hard to pin down.
I'd expect with both machines patched that clock for clock AMD are slightly ahead.
I've got a 2700X at home and it's a monster and I'd Zen2 is on the conservative end of the leaks/rumours it's going to be a total killer.
And no, Intel is not unique in speculation capability.
For large codes there is just no getting around the fact that the AMD parts have a weird and highly fragmented cache architecture, very slow memory access, NUMA memory latency even on the same socket, and a frankly broken branch target buffer. Xeon has none of these problems.
If by crushing it you mean performed nearly identically, and if by 5 years old you mean the very latest E5 chip Intel offers[1], then yes.
Also, they were using EPYC through a cloud VM provider, which means not only is there the typical overhead of Xen, but also possibly some aggressive Spectre mitigations. Their Haswell machine was likely bare metal.
[1] Because E5 and E7 chips only migrate microarchitecture every few generations. But since Intels fab headaches they're actually stuck nearly 4 generations behind, rather than the typical 2-3. Only desktop and E3 series chips use the latest microarchitecture.
EDIT: I forgot they changed their naming scheme. The latest 56-thread capable Xeon Platinum chips available last year used Skylake. I was confirming my understanding with https://en.wikipedia.org/wiki/Xeon but didn't see this page: https://en.wikipedia.org/wiki/List_of_Intel_Xeon_microproces...
Intel's performance hasn't changed for 3 generations, either. They've been stagnant for years now with their 10nm roadmap being super late at this point, delaying their architecture updates at the same time.
I still disagree with the previous poster's claims and conclusions, but I really should've done my homework better. I tried but clearly not hard enough.
POV-Ray doesn't fit in L1 and Epyc 7601 trivially bests the Xeon 8176, and does so while using a ton less power. Similarly NAMD, even with AVX & compiled with Intel's ICC, runs waaaay faster on Epyc 7601 than it does on Xeon 8176.
Maybe your workload is exclusively MySQL or looks a lot like it. If so, sure, go buy Intel. But there's a ton of cases where there is no where close to such a gap, including large cloud workloads. You keep taking a single benchmark result and massively over-extrapolating it to mean the entire cloud hosting world. That's not how this works.
What does this comment refer to? As far as I know AMD is in pretty good shape.
I wished they have made the move earlier, like start of this year, but it is now all set for Q3.
I have high expectation but I trust Dr Lisa Su and her executive team. And Intel is currently having a new low in my book.
I've read that wafer yeilds on zen2 are 70%+ which gives a huge advantage on cost and production efficiency to AMD. I think Intel's skylake yield for an equivalent 28 core die is sub-40%. If anything the limiting factor is going to be TSMCs ability to give AMD capacity on it's very crowded and in-demand 7nm fab.
In a few years, if Intel have their act together and AMD start having more vulns, then we may swing back the other direction. Only time and vendor actions will tell. This assumes we even move to AMD.
But I believe you're right in that those exploits are not as dangerous in a non-shared hosting scenario.
An attacker could carefully craft network packets to force the control flow of existing software to manipulate the CPU state. They could and then use the timing differences in network packets to read data out.
Would be painfully slow, but theoretically possible.
But if you're just running a local home system the chances of being targeted with such an unreliable attack are virtually zero.
I think that's the catch: it's the set of cases where someone can get code execution without being able to directly retrieve the targeted data, which is already pretty low, and where the best mitigation technique is to harden against these attacks rather than use a dedicated service (e.g. if you're using an HSM for crypto or have a dedicated SSL-offload box, you don't have to worry about keys being compromised when someone finds a bug in the much greater exposed surface area of your application).
So, if you have a system which you cannot expose it is a pretty big damper on performance.
If you're not sharing your hardware with other code you don't trust, it's not a problem.
The list of sites I have whitelisted JS for is extremely short.
What code do you actually trust? (to be incorruptible/unhackable, etc)
When I install software I am implicitly trusting it, whether that software is trustworthy or not. If it is exploitable that doesn't mean I didn't trust the software. On my personal computer, the expectation is that only the software I have implicitly trusted will be running. If I install something that isn't trustworthy but trust it anyway it's game over. If it's exploited, that's a different issue to trusting it.
Shared VMs are a totally different ball game. All of a sudden I am sharing resources with a potentially bad actor.whethrt or not I trust them is kind of indifferent, I don't get to choose who I colocate with, and ultimately that's the difference.
Being pedantic about this is not helping anyone actually be more secure: the odds are orders of magnitude greater that someone will be compromised by a different security problem than by the diminishing returns trying this attack against a current browser, and that the attacker will directly obtain the data they want without needing a side-channel attack.
As for app stores there's typically easier malware attack vectors if you manage to get an app installed & past the scanners in place with that typical back & forth escalation game. And those run in process-level sandboxes with clamped down syscalls. You can't exploit a spectre BPF attack if you can't use BPF at all due to selinux rules, for example.
Moving past consumer what else are you really going to attack? Sure you could rent some virtual hosting somewhere and hope you get lucky attacking a neighboring VM, but what's really your end goal there to justify the time & risk? It's not like you can reliably pick a target with that, you just have to go fishing and react to what you may or may not find.
However, when a vulnerability is found, then spectre etc. make it easier to abuse that vulnerability to do something useful.
I upgraded a VMWare ESXi server hosting FreeNAS from a consumer motherboard with an Intel Core i7 3930K and it was completely painless.
[0]: https://www.asrockrack.com/general/productdetail.asp?Model=E...
My current HP home-server has been with me for ~10years
I've been seeing people try to make power architecture (ibm) servers a thing for 12+ years now, it never happens, because ordinary developers can't afford them. Compared to what you can put under your desk for a thousand bucks based on any Intel or amd, amd64 architecture system .
Or this? https://www.raptorcs.com/content/TL2B01/intro.html
$2700 and probably is outperformed by a $150 ryzen cpu on a $110 motherboard.
Except that it won't have open firmware on a CPU hardly anyone targets. Whatever you build will probably be vulnerable. Unless you're running OpenBSD or something.
In case you are curious to see benchmarks.
AM4 boards have similar verbiage.
They do not mention unregistered DIMMs being supported.
Obviously EPYC has a place but for the home usecase you could use a Ryzen as a substitution for Intel Xeons because of the baseline features of Ryzen.
Threadripper has 64 lanes, though, which is definitely a lot.
The future is more likely ARM- I know a few companies are experimenting with moving their internal servers to ARM, and AWS started offering that platform a few months ago.
But I'm not sure how long this takes, people have also been wrongly predicting the death of x86 since the 90s.
We heard the very same argument about Microsoft Windows back in the 90s and 00s. "Linux [1] just wasn't well tested." It is purely a hypothesis, without any proof whatsoever.
Microsoft lost its dominance. Intel not yet, but this could lead to that. The question with losing dominance (e.g. being monopolist or market leader) is always a question of when, not if. And if it happens some people will lose a lot of money [2]. There's going to be a time where the USA is no longer #1 world power. The question isn't if, it is when. History teaches us this much.
[1] (Whatever that means.)
[2] (Or "money".)
Fast-forward to end of 10s and the dominant client is the web browser stack where Microsoft is barely relevant. That Microsoft Windows and Microsoft Office are still dominant is hardly important as the market trend is that these markets themselves are less relevant. If not merely for the fact that venfor lock-in has shifted, in favor of the other 4 in the FAANG stack.
1) AMD knew about the potential vulnerabilities that would have emerged by making the CPUs faster "Intel-style" and therefore said something like "no we won't take those risks", or...
2) if it was just by pure luck, or...
3) if for them that option wasn't technically feasible,or...
4) if maybe they did not have as "good" (relatively speaking) engineers as Intel.
Personally I don't think that it could be #1 as I think that the pressure to make the CPU faster (and therefore being able to better compete with Intel) would have been a lot higher than to say "nah, this might hit us in the future so let's opt for the safe variant".
Intel took a gamble and lost. Well, they lost from an honest engineering standpoint. Still unclear if they lost anything from a market perspective, and we may never know because they're losing even bigger in terms of fabrication, which will obscure the effect of their horrible security fumbles.
[1] You don't need to know the details of an exploit to avoid a vulnerability. You just need to know what you don't know, and for side-channel attacks it's relatively easy to know what we don't know--any calculation where a timing, power, etc differential is even indirectly visible (at any level of precision) is suspect. Sometimes you have to take the risk, but some risks (i.e. speculating across security domains) are just too great, especially in an environment as sensitive and security critical as a CPU. Intel engineers knew. They couldn't have not known; they're hardly incompetent.
Also, something everyone seems to forget: All of these cloud providers are running Skylake chips, which were launched in 2015. Google Cloud in particular will even give you chips OLDER than Skylake by default if you don't specifically request Skylake. Even assuming instances like an AWS m5a are running on Zen 1, not Zen 2, that's a 2017 architecture.
So you're presented with all these graphs that say "AMD only lost 5%, Intel lost 25%, fuck Intel" but the reality is that Intel was previously far faster than AMD, and they're not even fabbing their best designs for the data center. Intel definitely had more vulnerabilities and they WERE hit harder, but its more nuanced than just blindly wondering why more cloud providers aren't making a fleet-wide switch to AMD.
Not so surprising now that we know Intel merely skipped a lot of checks to get there though. I mean a lot of the vulnerabilities impacting Intel only have been the likes of "when predicting, the processor does it even it shouldn't to avoid the delay from checking".
Is it really fair then to say that AMD doing it the proper way was slower, or that the loss of performance of Intel can't be used as a way to congratulate AMD on being safer on that front ?
That's a technically true but also misleading statement. It is exclusively on heavy AVX loads that any such delta appears. If you're not using AVX, then the single-thread differences are ~15% or less. So Intel losing 25% does now potentially put them behind on most single-threaded workloads.
My understanding from a Google blog that talked about it indicated that Google felt like almost any speculative execution... is a risk. So while there might not be someone exploiting it now, they considered the practice potentially an issue. Accordingly future performance will possibly still degrade later as further changes are possibly needed?
Every company with high performance single thread designs has out-of-order with speculative execution designs, ARM, AMD, IBM POWER and mainframe/Z, Intel, MIPS, and SPARC. RISC-V is working on it, with the Berkeley Out-of-Order Machine (BOOM) and perhaps other cores.
And doing research I should have done a long time ago, SPARC V9 has Spectre vulnerabilities (https://www.zdnet.com/article/meltdown-spectre-oracles-criti...), and MIPS made a statement that a couple of their designs were possibly vulnerable (https://www.mips.com/blog/mips-response-on-speculative-execu... and this followup assumes that: https://www.mips.com/forums/topic/mips-mitigations-for-side-...)
The other vulnerabilities: Meltdown, Fallout, the recent MDS attacks (RIDL, ZombieLoad, Store to Leak forwarding), are Intel specific because they’re caused by the way Intel specifically chose to skip/defer security enforcement checks in parts of their implementation.
Other companies didn’t do this.
ARM, and IBM POWER and mainframe/Z also did that (Meltdown): https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili... ARM also has a Rogue System Register Read vulnerable design (https://developer.arm.com/support/arm-security-updates/specu...).
AMD's server market share is minuscule and dropped as of 19Q1, 3.2% to 2.9%, although healthier and growing smartly in desktops and notebooks, up to 17.1% and 13.1% as of last quarter (I would guess the two are related if the comments made in this discussion about their being fab capacity limited are correct). That means they're less significant targets for researchers than Intel and ARM. We might also assume AMD has less manpower to devote to finding vulnerabilities than Intel has.
These latest Intel vulnerabilities, Foreshadow/L1TF and this week's? They're all targeting Intel specific details, for example the first Foreshadow version targets the SGX enclave. See also ARM's Cortex-A72 Rogue System Register Read (RSRE), Spectre Variant 3a vunerability: https://developer.arm.com/support/arm-security-updates/specu...
The odds that AMD specific features have vulnerabilities than simply haven't been looked for yet is very high, their Spectre vulnerabilities show that they too generally got caught with their pants down.
By contrast having to deal with the issue in software or worse, by turning off SMT, is much more damaging and gets you the 10s of percents of performance impact that people are talking about.
Go with Intel and risk losing a huge chunk of your performance some day, or go with AMD and know what to expect. Different people and companies have different risk tolerances, so it's not an obvious choice one way or the other.
From Zen thereon, they've been more or less on par in single threaded performance, and have the performance edge on multicore performance, while being cheaper and unaffected by most of these new CPU vulnerabilities, on top of that.
Rowhammer is arguably more aptly described as a defect; in that case the RAM is not behaving to spec.
I certainly didn't see one when booting up a new Intel computer.
Intel's hardware still provides the the same MHz. So the analogy would be more like due to safety concerns, a software update to the car increases the braking used by traction control.
That feels charitable to Intel, but unless they knew about the exploit, they used the tools in their arsenal to maximize performance. Now they are reducing it for safety.
Maybe it's more like how auto manufacturers are now having to increase wind resistance by designing pedestrian friendly hoods. Obviously that's not "after purchase" but I can't think of a better analogy!
In that spirit, here is a different car analogy for you.
This is like buying a new car, and then having the odometer suddenly jump by 150,000 km: you paid for new, but are stuck with old.
Or, we could go by features. Cars get new features. Suppose you pay for a car with various bells and whistles: on-board Wi-Fi, navigation, semi-autonomous driving and whatevernot. But then most of it doesn't work, so your car is like something from 2005. Forget bluetooth; just 1/8" aux jack, or CD player.
If you follow the semantic descriptions given in their architecture manuals, can you infer these insecure behaviors?
You're paying for a new one, but it performs like something that is several years old that you can obtain much more cheaply.
Another aspect of it is consumers who chose an Intel chip over a competitor's. Intel does promise competitive performance. Their pitch isn't "we are slower, but just buy us on the brand name alone".
These intel mitigations impact those services just like everyone else.
This assumes the servers for S3 etc are dedicated to the task.
I’m not sure that follows. Unpatched, most of these Intel CVEs are almost like unpatched local privilege escalation vulnerabilities. Once you’ve compromised a low privilege process you can sniff your way to more and more powerful credentials on the machine, host, and in the network.
Unless higher abstraction cloud service providers are dumb enough to just run everything as root, in which case this doesn’t change anything, they probably can’t (competently) get away with not patching these. Defence in depth matters.
Interesting take: It's better to offer it as a "one click app" aside a basic cloud (docker/kubernetes) platform if you don't want to suck these costs.
So to the extend that you're suggesting that cloud providers are happy, or may even have had a hand in this: nahh.
I wonder if I can skip the performance penalty for my own (hardware) server too... But I am not smart enough to answer this question myself.
Most people don’t. It’s a defence in depth thing — if someone hits an exploit in a service you want it to be running the lowest privileged user it can get away with so that it can’t rapidly pull all the data out of other subsystems, roam out into the local network and dump DBs, etc.
With these vulnerabilities unpatched, an attacker running code as the low privilege user account can sniff credentials for other accounts on the machine, or underlying host, private keys that authenticate against this machine and therefore probably others in the network, etc.
I’d say it increases risk quite a bit.
> The performance loss is not visible for our customers, we have to manage the loss in ourselves. It’s a kind of hidden defect.
We've seen maybe a 2% hit from spectre & meltdown mitigations. It's hard to even tell because it's a small enough amount that it tends to get lost in the noise.
Imagine a slowdown (performance penalty) of 50%. It would double the time to complete a task, thus doubling the cost (expressed in cost increase that would be a 100% increase!).
Generic formula: 1÷(1−x)−1
For x = 25% ---> 1÷(1−0.25)−1 = 33% duration (cost) increase
Now it's paying dearly for that mistake, with up to 40% performance loss on Chromebooks due to the disabling of HT:
https://www.techrepublic.com/article/mds-vulnerabilities-lea...
Google broke one of the most basic business rules: never rely on a single supplier. You're always worse off in the end, even if the exclusivity deals seem very tempting in the short-term.
[1] https://cloud.google.com/blog/products/gcp/compute-engine-up...
Weren't Chromebooks on ARM before x86?
> Google broke one of the most basic business rules: never rely on a single supplier. You're always worse off in the end, even if the exclusivity deals seem very tempting in the short-term.
I don't think this is universally true, although I agree that it's probably prudent. Diverse options/suppliers are a risk mitigation, but they do have a cost.