Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower
phoronix.com
phoronix.com
We new that speculative-execution mitigations would sometimes reduce performance, and evidently they do. :-(
pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier
1. Development machines should be configured identically to production machines. (Do you install a GUI and development tools on your production servers?) Occasional differences in behavior between development/production are par for the course, and is why staging environments are commonly used.
2. The mitigations affect the execution result of code likely to be developed/executed at a typical software shop. AFAIU the attacks are timing-based, and won't affect valid code that's not specifically looking to exploit them.
Pretty much any application where there is limited execution time budget. If it takes too long it's considered broken, unusable, undesirable etc. Any realtime-ish stuff.
Ad bidding, video processing, audio processing / messaging, industrial control systems, robotics, general signal processing, logging systems that will now be overwhelmed and drop messages, databases that will experience timeouts and retries, scientific computations that will now take weeks more to run and potentially screw up other projects.
But hey it's a great time for Intel. Hey, psst, over here, I got a faster later gen CPU for you, for an easy price of $999.99 to bring you performance back to where it was last week.
https://github.com/torvalds/linux/blob/master/Documentation/...
modinfo ${somemodule}
man lsmod man modinfo
This is elementary level linux, you should know this.
Also the kernel flags, don't block intel microcode "improvements". You have to do this with:
sudo apt-mark hold intel-microcode
Mitigation isn't a fix, it's a bandaid.
If performance takes a hit by about 20% it is going to be noticeable. Next up I think people will start looking to upgrade to a faster CPU just to get back to last week's performance metrics. They might now consider AMD.
If AMD finds this affects them less they should sponsor security researchers writing proof of concept exploits to convince people that that turning off the mitigation is an absolute no-no, so it will for force people to take a performance hit and go shopping for CPUs.
From what I have read so far is seems that AMD CPUs have had the fewest vulnerabilities/slowdowns? But I can't be sure since I haven't seen a complete comparison (including these new vulnerabilities).
- The general Spectre family, which affects most (if not all) CPUs built with speculative execution.
- Meltdown and L1TF, which only affects Intel CPUs due to them delaying security checks until after speculation has taken place.
AMDs, ARMs, etc. that use speculative execution are going to be vulnerable to at least some variant of Spectre (there are 4 variants known right now). ARM published a table[0] explaining which of their CPUs are vulnerable to which variants. I'm not aware of any such table for Intel or AMD.
Microsoft published some interesting tables[1] explaining which mitigation protect against which Spectre variant, and under which thread models they operate.
[0]: https://developer.arm.com/support/arm-security-updates/specu...
[1]: https://blogs.technet.microsoft.com/srd/2018/05/21/analysis-... (scroll down for the tables)
"The Spectre/Meltdown Performance Impact On Linux 4.20, Decimating Benchmarks With New STIBP Overhead"
https://www.phoronix.com/scan.php?page=article&item=linux-42...
Would that hardware mitigation reduce this performance loss?
[1]: https://www.anandtech.com/show/13400/intel-9th-gen-core-i9-9...
A little Googling found an analysis from someone more knowledgeable than myself that backs this up: https://secure64.com/not-vulnerable-intel-itanium-secure64-s...
tl;dr Meltdown takes advantage of out-of-order execution, which Itanium simply doesn't have. Spectre makes use of speculative execution, which Itanium only has in a version too limited to support the attack.
I'm not an expert in this field so I can't attest to the credibility of this analysis. The Spectre part in particular sounds a little hand-wavy but it might just be over my head. But it aligns with my intuition, which is that the simpler and more explicit architecture doesn't have as many places where data can accidentally end up.
Can someone explain (or link to an article) how a tweak to HT branch prediction heuristic can have such a huge impact on performance?
The problem is in the mitigations necessary to make it impossible/more difficult to exploit these side channel attacks. And that is costly because memory and needs to be moved around constantly. So that adds a ton of extra overhead whenever a context switch is made.
I can test it once I get the new firmware.
aren't vtable calls / function pointer calls indirect branches ?
but if their cost increases by a large factor... besides, in any large compiled program, the core would certainly be based around some kind of programmable pipeline, and these would generally be implemented like this unless they wrote their own JIT compiler.
I didn't follow your comment about a "programmable pipeline". I don't think many or any of the Phoronix benchmarks are based on a pipeline with indirect branches at their core.
I think a bunch are. e.g. for instance FFMPEG / libavfilter which is basically a node graph set up at runtime. Don't know for cinebench since it's closed source, but Blender present in the benchmarks is also based around a nodal rendering architecture. Stuff like PHP / CGI also heavily depend on function pointers for their behaviours - PHP with its plugin architecture, and CGI where all web requests go through FPs : https://github.com/php/php-src/blob/master/main/fastcgi.c#L8....
So I still doubt that that indirect branches are "dense" in those benchmarks: it just doesn't make sense since the core work they are doing are highly tuned encode/decode/whatever kernels, even if there is a control layer over top of that using indirect branches.
> But why is it slower? More work on f&#!#(# Spectre!
Running other peoples code (on other peoples machines) is a cloud thing, but they have an iron grip on the kernel and other software.
Instead, everyone has to pay the performance tax proactively, regardless of their individual threat model. That is not how this sort of thing should be handled.
I'd also personally not feel comfortable using a Windows PC, given that running unchecked 3rd-party code is sort of required to do anything with it.
<terms of lines of code changed with more than 354 thousand lines of new code added at the end of October when this merge window opened.
The question I was responding to was: “accepting that, rightly or wrongly, Linux drivers run in kernel mode, does it make sense for them to be in the source tree?”
The only real problem I've had has been with soft lockups, which I was able to solve by disabling a power saving feature on the CPU. If AMD doesn't mess things up, they can really catch up to Intel in the next couple years while Intel is working on fixing their issues.
Now I assume it's because while even if the textual content is quite small, all the images mean there's quite a bit of vertical length to the article. I had to scroll down quite a bit to get to the next page button. The second page is quite small though, so maybe a better page break algorithm in is order. e.g. break at size X unless total length is X*1.25, and if only two pages, try to make them somewhat even in size (erroring on the side of a larger first page).
If the choice is between unsecure but fast to secure and slow, the default should be fast. Let applications who care about side channel deal with it their way, why the fuck should everything be so slow just so that few apps which actually care about side channel will be secure?
The only reason these attacks even have the appearance of being unpractical is because there are so many other areas of our computing systems that are even easier to attack.
The only reason we are in this current Spectre et. al. mess is that CPU vendors choose to prioritize performance over security, and we now have to live with their bluff getting called for another hardware generation or two. I suspect that the performance impact will be far less when the fixes are done during the hardware design process, instead of figuring out how to bolt them on after the fact.
Ultimately, I think we are overdo for a new computer architecture (both in terms of security and performance, we are being encumbered by the need to keep new architectures backwards compatible with old ones).