iPhone update for Meltdown-Spectre: before/after performance benchmarks | Hacker News Reader

iPhone update for Meltdown-Spectre: before/after performance benchmarks

melv1n.com

157 pointsjustdutch109 comments

This article doesn't make any sense. The author is doing generic phone benchmarks. But the Spectre fix is a webkit fix only, not an OS-wide fix. If they're seeing performance regressions across the whole OS because of fixing Spectre, something's seriously wrong with their benchmark methodology.

Edit: The author upgraded from iOS 11.1.2 to iOS 11.2.2. This isn't just a test of the Spectre fix. The most likely explanation here is upgrading to iOS 11.2 caused their iPhone 6 to start throttling due to battery wear (11.2 added throttling to iPhone 7, and it's plausible that it changed the conditions for throttling on iPhone 6). It's also possible that this is instead caused by the Meltdown patch, but these numbers are still way out of line with what was expected for Meltdown on iOS, whereas they're very much in line with what we've been seeing with battery throttling.

What sucks is there is no way to fix Spectre without going the battery throttled version.

Yeah there is. Get your battery replaced. Apple will replace your battery for $29 (as long as there's sufficient wear, but if you're seeing throttling, you're well past the cutoff).

>as long as there's sufficient wear,

That policy has changed, they will replace your battery upon request for $29 no matter what their diagnostic tests say.

What's your source on that? Apple's message¹ said

> Apple is reducing the price of an out-of-warranty iPhone battery replacement by $50 — from $79 to $29 — for anyone with an iPhone 6 or later whose battery needs to be replaced, available worldwide through December 2018.

The "whose battery needs to be replaced" sure implies they're still going to actually test it.

¹https://www.apple.com/iphone-battery-and-performance/

https://www.theverge.com/2018/1/2/16840400/apple-iphone-batt...

This.

The article should be flagged off the front page or edited by the mods to show that it's questionable.

This is a single person's test results, across versions that are known to have touched the battery/CPU slowdown logic, showing insane performance degradation across almost every aspect of the phone... it's super questionable. And right now it's #1 on HN despite all of that, just spreading misinformation.

I just did a 11.2.1/11.2.2 benchmark of my iPhone 7: https://browser.geekbench.com/v4/cpu/compare/6303880?baselin...

The result is that 11.2.2 is slightly faster than 11.2.1 (around 2%).

And here’s one of my iPad Pro 10.5”: https://browser.geekbench.com/v4/cpu/compare/6304146?baselin... (0.9% increase in performance).

I did this same test with an iPhone X and iPad Pro 10.5" as well as a several year old Mini 2 and didn't see a difference. In fact, I saw what you did, my scores got better in most categories.

I have an iPhone 6, and didn’t notice any further performance issues after upgrading to 11.2.2. I did have pretty serious issues (which have mostly remained) since upgrading to iOS 11. So I don’t think the latest version did anything more, at least on my phone.

>> a significant decrease in performance on the iPhone 6 up to 50%

Something's up. I updated my 6S yesterday and have noticed zero in performance changes or battery loss. Still plenty of bugs though, I was reading an email and the 'flag/file/trash/reply/new' bar totally went away. At least the touchscreen hasn't gone unresponsive, causing me to have to hit sleep/wake to toggle it back on. Maybe they finally fixed that.

same behaivor here... no perfomance / battery hit.

6 or 6s?

I have a 6S+. I noticed a performance gain going from 11.2.1 to 11.2.2.

Before: 2384.

After: 2541.

6 and 6S have different CPUs.

On an iPhone 6 - I actually gained performance from 11.2 to 11.2.2 (I neglected to run Geekbench on 11.2.1). Previously my best single-core score on iPhone 6 was 1536 (and only when battery was above 93% would I get in the 1500s), now today I got 1540 single core on only 66% battery - previously anything below about 70% battery would give me a score down in the 1200s.

There really seems to be some performance issues with either the network stack or the threading/background processing in iOS11 on older devices.

I've had periodic slowdowns and unresponsiveness with my iPhone 6 since upgrading to 11, but a few weeks ago I was downloading a bunch of music for a trip and my phone was virtually unusable until the downloads were done. Since then I've noticed unresponsiveness around the same time I see notifications for podcast updates or after exiting an app that was downloading tons of content (e.g. The Weather Channel app).

Ok, I'm sceptical about the results. The reason is that there doesn't seem to be a massive difference between the tests. Since this fix is about speculative exec, why would it affect crypto code which is very register based and branchless as much as sqlite which is full of branches and memory/storage based? Why would it affect AES which is hardware accelerated as much as integer processing which is not?

I'm not saying this is impossible - maybe there's something that I'm missing. But it just doesn't add up at the moment. I'd love a more detailed / repeatable test.

Is it possible that this benchmark has coincided with a battery that's just crossing the threshold of battery-throttling induced slowdown?

The low scores in the article are in a similar ballpark to my (slow) iphone 6 on ios 10.3.3 with a 3yr old battery - i.e. about half what they should be with a fresh battery.

I can't speak about these benchmarks (the site is down). But Meltdown is not about speculative execution, it's about out-of-order execution. The fixes for Meltdown involve getting kernel data out of user virtual memory, and flushing the TLB if the processor does not have the pcid instruction. So context switches - including system calls - will tend to be more expensive, and if the processor does not implement pcid, way more expensive.

See https://meltdownattack.com/. I haven't yet read the Spectre paper, but I can say that the Meltdown paper is very readable and walks readers through everything. I anticipate the Spectre paper is as well.

This is about Spectre, not Meltdown, which is about speculative execution. From the article:

> Apple released iOS 11.2.2 update to address Spectre security issues.

I'm not sure why Meltdown was added to the HN title...

The A8 doesn't have a dedicated crypto engine like Intel chips do (uncore I think they call it). The A8 does have a couple of AES specific instructions so each core can speed up the work, but they were executed speculatively and presumably are no longer done so.

From Geekbench knowledgebase: "Geekbench will use AES instructions when available, and fall back to software implementations otherwise."

I expect this test to use them.

These benchmarks are between iOS 11.1.2 and 11.2.2, which include the battery throttling code in 11.2.0.

We need a benchmark between 11.2.1 and 11.2.2 instead.

I've went through Denial, Anger, Bargaining, and now in an Acceptance stage after taking up to 31% performance hit on some of services managed by my team. Worst case has been Elasticsearch so far with our load pattern, taking that 31% hit.

Oh well, too bad, enjoying the ride.

Time to switch to an Epyc host?

We're actually using those new 24-core Epyc hosts for our data science pipeline (HDP/Spark cluster) and they work much better than our previous generation Xeon-based builds.

Unfortunately, with those things it takes plenty of time to re-evaluate hardware + in our case we're mostly in the cloud where Skylakes on GCP is the most we can get.

AMD is vulnerable to Spectre and are not fixing variant 2 because it’s “near zero risk”.

By near zero risk they mean that no one has yet reversed engineered their branch prediction which is required to effectively exploit it.

Give it sometime and the BTI patch would have to be enforced on AMD CPUs also.

What about using dedicated hardware and not running the patches? At attacker has to run malicious code on your server to exploit the vulns anyhow, right?

Sure, but I think that would (potentially, with the proper exploit) turn any non-root-level code-execution-able breach of your machine of a machine into a root level one, no? Maybe that's okay?

I usually don't trust people. More than one have we found things improperly secured and/or exposed to the outer internet, or otherwise at risk. We also run black-box software from third party vendors that provide compiled JAR files which is directly related to this fuss. Last thing I'd like to have is another attack vector at home.

How did you measure the performance hit? And what version of Elastic did you test?

I made similar benchmarks recently on my iPhone 6S, running iOS iOS 11.1.2 vs 11.2.1, before and after replacing the battery:

https://twitter.com/enstyled/status/948892447765131264

TLDR: It's 11.2.1 that is throttling the older iPhones, because of the battery wear.

Sample size of 1, but I Geekbench'd my iPhone X before and after upgrade.

11.2.1: Single-Core 4137, Multi-Core 9315

11.2.2: Single-Core 4039, Multi-Core 9876

Anecdotally as well, I haven't seen a noticeable difference in performance. So your mileage may vary substantially based on what device you have.

I just ran GeekBench 4 on my iPhone X on iOS 11.2.2 vs 11.2.1.

Single core: 4239 vs 4241

Multi-core: 10081 to 10203

So no difference.

The test done here was comparing 11.2.2 vs 11.1.2

Isn’t 11.2.1 the battery patch?

You are correct, it appears to be that "patch".

Source: https://discussions.apple.com/thread/8211392

So basically, the difference in performance is likely almost entirely due to the battery patch and has nothing to do the Meltdown-Spectre fix.

I wonder what the difference is between the 6 used in the benchmark in the article and the iPhone X that would cause these different results. They both support speculative execution.

Either way, encouraging results to a fellow iPhone X user.

A wild speculation: the newer procs have something similar to a pcid?

Does GeekBench 4 make a lot of syscalls? My understanding is that these are the most affected by the mitigations.

I'm curious what the delta is between perf hit on an iPhone 6 and an iPhone 8. I'm guessing that, given the deadline apple had to get this out, most of the attention & optimization went to current devices versus near-EOL devices like iPhone 6.

Is the Samsung S8 CPU also affected? I saw that some ARMs are not affected.

EDIT: It appears not the be on the list of affected ARMs [1]. The S8 Exynos 9 Octa 8895 is based on the Cortex-A53

[1] https://developer.arm.com/support/security-update

EDIT2: The Snapdragon 820 and 835 do appear to be affected which are the CPU in the US version of the S8

Assume that [EDIT: high-end - dogma1138 is right] smartphone-class processors and above are affected. Raspberry Pi and microcontroller-class ARMs are fine, though (but much slower!)

Most of them aren't affected mid range ARM and below aren't vulnerable, basically if you have a flagship product you should worry.

If we take the snapdragon example the 6XX's aren't affected while the 8XX are.

It looks like the Exynos 9 Octa 8895's 4 main (big) cores are a custom Samsung architecture which is almost certainly out-of-order, and is probably affected. The "little" cores are Cortex-A53, which is in-order and not affected.

Google cache:

http://webcache.googleusercontent.com/search?ei=jXNWWojdC4vc...

This might sound dumb but.. with these Meltdown + Spectre bugs would an attacker be able to penetrate to gain full access only if a user downloads a native app or would it be possible through the browser/js?

As far I remember JS exploit is possible. That's why Mozilla patched Firefox already.

This post, which I saw on HN yesterday, may answer some of that for you:

https://webkit.org/blog/8048/what-spectre-and-meltdown-mean-...

Short answer: yes.

Not if their browser hasn’t been patched.

Spectre can give you read-access to the browser process via Javascript, yes, as they described in the original article: https://spectreattack.com/spectre.pdf (section 4.3 on page 6). No write access, or access to the OS internals.

Haven't seen much coverage on how this affects game consoles like the Xbox and Playstation (which use AMD CPUs). Does anyone know if they've talked about patching it and how this would affect game performance (which is a pretty big deal for gamers who expect a consistent experience)?

Mike Ybarra has said there's no need for them to patch the Xbox One against these attacks:

https://www.trueachievements.com/n31065/xbox-one-consoles-un...

I will repeat these benchmarks on my iPhone 6 when I get home tonight...but in the meantime, the benchmark numbers in this article look almost exactly like the ones I had as a result of battery throttling. I bet the actual patch-related performance hit is pretty minor.

I’ve more than a little skeptical of these results, I too am very disappointed in major CPU vendors (especially their PR and management teams), but these results seem more than a little smelly.

I am what I’d consider a heavy iOS user on multiple devices and I don’t believe I’ve honestly noticed any difference at all. If these results were correct, I believe I’d notice at least a 10-15% decrease in performance - but no. What I haven’t looked at is battery life so I cannot comment on that.

I wonder how much extra revenue this will bring to AWS/GCP etc selling more instances to cover the performance loss on servers?

sadly, this is the first thing I wondered when all the news came out.

It's energy providers that will win the most.

All major cloud providers are buying renewables for their operations. Yay?

At least the additional renewable generation capacity built out will exist for decades (while processors should become more efficient as these security issues are addressed and new hardware rolls out).

>All major cloud providers are buying renewables for their operations. Yay?

Not yay. This is an overnight massive bump so they are going to consume from the same existing pool of renewables, forcing utilities to use more non renewables to make up for the demand difference.

Also, even when they do build out the renewable generation to make up the difference, it's still environmentally a problem due to the manufacturing of the renewable generation (solar panels, windmills, transformers, etc).

We have no surplus of renewable energy to absorb this kinda stuff so more energy usage is always bad at this point. It's just slightly less bad when they commit to paying to fund renewable.

I've read a few articles saying that there is so much Solar adoption in the U.S. that there is a surplus at some times (summer days I believe). It use to be that there was too much use from home AC systems so power was expensive during the day. It seems to be switching to night now as so much is generated during the day (with just 3% adoption of rooftop Solar).

Here's an example of one article on the matter, which I just skimmed a little.

http://www.latimes.com/projects/la-fi-electricity-solar/

I'm guessing that CPU's use a whole lot less electricity than things like AC systems, so they probably don't affect total electricity used by that much. I was able to surmise that a 3 ton central air conditioner uses around 3500 watts per hour while an Intel I7 (whole system) uses around 150 watts per hour.

My theory, then, is that this won't affect total power use in the U.S. all that much.

It’s a hit on AWS users, that means it’s also a hit for Amazon themselves.

They’ll sell more servers, but they just lost 10% (or whatever) of their headroom all around the world.

I'm guessing more than 1/2 of all AWS instances are idle. Probably 10% are systems nobody knows anything about, whoever started them left the company..

Add the 5% that are nothing but public garbage bins for personal information collected by various companies and intelligence agencies, just waiting to be discovered!

tested with GeekBench 4 on iPhone 8 Plus if anyone interested: - iOS 11.2.1: Single-Core 4257, Multi-Core: 10187 - iOS 11.2.2: Single-Core 4259, Multi-Core: 10287

Looks like no performance impact at all.

My iPhone X has been on 11.2.5 since the day it came out, and I haven't perceived any slowdowns.

Have not done any benchmarking, and I'm not saying performance is the same as before, but anecdotally I haven't seen a difference. Just one guy's opinion.

Do you game or do anything CPU intensive?

the latest iOS version as of today is 11.2.2

I don't think 11.2.5 exists

It has existed in beta for a while.

https://www.macrumors.com/2018/01/09/apple-seeds-ios-11-2-5-...

I just updated my 6+ and ran Geekbench before and after. No major difference in scores and the single and multi-core benchmarks are consistent with the comparison numbers for other 6+ phones. Battery is in good health.

this leads me to a couple questions, hopefully you smart folk can answer:

1. I thought Spectre was "Intel-only", and Meltdown was the general case, which is less severe but effectively nearly everywhere? If so, how is an iPhone susceptible to Spectre?

2. Beyond that, I thought meltdown/spectre was an x86 problem. So why all this trouble on phones, with ARM?

3. I've read the first, simplest variant of meltdown, and it is so beautifully simple. Is this "speculative execution + cache timing" thing an entirely novel exploit, or have we seen incarnations of this before?

1. is reversed: meltdown is intel-only. Spectre is a more generic speculation attack.

2. Apple's custom ARM chips have extensive speculative execution, and clearly their fixes for it are extremely expensive.

3. It's novel, which is why security researchers are so excited! Branching based side-channel attacks were previously only known to be able to track execution (so branch-based AES encryption was vulnerable to having its keys stolen)-- but reading arbitrary data is much stronger.

Re: #2 - it’s a speculative execution issue, not x86 specific.

1: Spectre is the general case, Meltdown is the Intel-only evil cousin.

2: Spectre applies to all modern processors with speculative execution, which includes smartphone-class ARM processors. Raspberry Pi's and microprocessors ("toasters") are not affected.

3: There's some prior work - Spectre didn't fall from thin air - but using speculative execution as the basis of an exploit makes Spectre the first bugs in, I believe, a new class. (Meltdown, on the other hand, is just a silly mistake that shouldn't be repeated.)

Can we get a before/after with a new battery vs a battery with 1k, then 10k, cycles on it?

I benchmarked my 8 Plus yesterday, updated from 11.2.1 to 11.2.2.

No differences noticeable under same circumstances, maybe even slightly faster (Compute benchmark, Geekbench 4).

Something isn't right with his benchmarks.

My iPhone 6 benchmarks:

Geekbench test taken on Dec 18, 2017 with iOS 11.2

Single Core: 1566

Multi-Core: 2697

Geekbench test taken on Jan 10, 2018 with iOS 11.2.2

Single Core: 1551

Multi-Core: 2675

Percentage slowdown:

Single Core: 0.96%

Multi-Core: 0.82%

Do you have Geekbench test with iOS 11.1.2 instead of 11.2?

Shouldn’t be needed if you go by absolute numbers - OP had 9xx instead 15xx .. so, clearly something is not right

Can't wait to see before/after benchmarks for Android. Something tells me it's going to be even worse than iOS.

Depends on the phone. Lower end phones use cores like the A53 which don't have enough speculation HW to be affected at all.

This is...highly questionable. 11.2.2 only updates WebKit, which shouldn't affect GeekBench.

I keep getting bad gateway from some awful CDN, is there an alternative link?

That's Cloudflare and it's not sending the bad gateway, it's telling you that the upstream (the OP server) is returning a bad gateway.

Except Cloudflare is saying this: "However, because the site uses Cloudflare's Always Online™ technology you can continue to surf a snapshot of the site"

It shouldn't be telling you that the host is failing.

Aaand it's down. Anybody have a mirror?

archive.org does: http://web.archive.org/web/20180110193140/https://melv1n.com...

Why does Cloudflare tell me I'm able to browse a snapshot while the site is offline, but there is no snapshot?

I get the same, and have seen it for other sites recently.

I've never known Cloudflare to do things well. It's the last CDN I'd want to use.

Edit: For example, the 502 page is showing, but supposedly, you are supposed to see the site based on this: "However, because the site uses Cloudflare's Always Online™ technology you can continue to surf a snapshot of the site."

Yeah, this is the first time I've ever seen the "you can continue to surf a snapshot of the site" message along with the 502 though. Or maybe I've just never looked that closely.

This is because, humorously, the snapshot is actually of the 502 page. Apparently Cloudflare kicked in too late.

No, it's not a snapshot of the 502 page, because I see my own IP there.

Is that how is's supposed to work? Because any time this happened in the past to a site where it is overcrowded and the Cloudflare banner is showing, the 502 page is showing as well, so I thought it's part of the Cloudflare error page?

Is this a "bug" we can recover from (in future hardware or software) or are we going to collectively take a lifetime -18 hit to our Moore's Law progress and lose the gains from this technique forever?

At least for Meltdown there are CPU microcode updates out there that should mitigate it, which means the OS level mitigations can be rolled back. And I haven't measured it yet, but I suspect the microcode mitigates are faster.

Regarding Spectre, I think it's too early to tell.

I'd like a new computer architecture, where communication between different parts of software systems was more explicitly treated as communication. Instead of an exception to pass control to the kernel, how about sending data? Instead of multiple CPU communication through memory (and therefore the cache) how about sending data?

Wow. After ios 10.2.1 halved performance on my iphone 6 due to the battery issues, I next avoided 11.0 as I accepted responsiveness would be slower on the older iphone6 model.

Now if we want these security fixes that's another -40%... Yikes!

However, it looks like ios 11.2.2 and a new battery is still slightly faster than ios 10.3.3 and an old battery! (geekbench single/multi core score from article of 924/1616 vs my 844/1379)

I don't understand this logic at all. The fixes are being made for a reason... avoiding the update just means you are getting more performance for a less secure system. Are you betting on a law of averages to break in your favor? Hoping you're not a target of hackers? what am I missing with this strategy?

Just that I could be looking at a > 75% performance reduction on the same phone compared to when it was new due to the combo of battery degradation, security fixes and ios9 to ios11 general slow down.

I think that if they've disabled some cpu core features that might reduce the maximum power loading anyway and make the double whammy impact of security and battery related slowdown unnecessary.

The OP doesn't seem to suggest that they won't update, just that there is a considerable price to be paid in doing so.

> Are you betting on a law of averages to break in your favor? Hoping you're not a target of hackers?

Aside from anything else, I wouldn't be at all surprised if some people do take this gamble. They shouldn't, but if they're informed about it then it's in their own hands.

All of these issues require local execution. It’s perfectly fine for many single user, performance critical use cases to make this type of security trade off.

> local execution

Delivered via JavaScript

So don't browse the web from your server. Why are we updating servers?

Coz you share virtual machines where others run code.

It wouldn't be the first time that a buffer overflow somewhere led to arbitrary code execution. What we're patching is a mechanism that allows escalating from unprivileged execution to reading kernel memory, and a mechanism that defeats KASLR

Reply on news.ycombinator.com