Hacker News Reader

Comment by dak1 | Hacker News Reader

The story has been updated with the following:

"Apple issued a new, stronger (SHA-2) Mac App Store certificate in September, before the older (SHA-1) one expired, as planned. The new Mac App Store certificate was using the current, strong SHA-2 algorithm. However, some apps were running receipt validation code using very old versions of OpenSSL that don’t support SHA-2.

OpenSSL started supporting SHA-2 in 2005, which is why Apple didn’t foresee this issue."

View on HN

This needs to be upvoted, and the title changed. Turns out it was due to developers using incredibly old security code. Rather than an expired certificate.

Yup, this changes my own point of view from "Gee Apple what a blunder" to "Some developers can be really dumb".

Apple built a lame DRM system, tossed out some half-baked sample code, and then told third parties "OK, you guys go implement this yourselves" without even so much as providing a library to help with the hard parts. Then they made a pretty substantial change without even checking to see if all of these one-off third-party implementations could deal with it.

Given that OpenSSL has supported SHA-2 since 2005, and the Mac App Store was announced in 2010, then released in 2011, I'm really not sure you can blame Apple for thinking it'd be reasonable to expect App Developers to not be using 5 year old versions of dependencies.

Doesn't matter - if you roll out a service, your number one priority is to keep the users happy by keeping your service running. And there's something called testing too - it's not like it was rocket science for Apple to detect problem apps and get that sorted before the SHA-2 rollout. They are taking a 30% cut for crying out loud.

They could even do something really crazy like telling everybody about the change in advance so third-party developers could double-check their code and make sure it still worked.

Yes! Communication is the first measure to try before anything technical as a second line of defense - but that's not a part of Apple's DNA :)

The expiration date is part of the certificate description. Events like this happen day-in, day-out. "Telling everybody" is not the solution, comprehensive testing is.

The problem wasn't simply the expiration date, but also issuing new certificates with SHA-256 instead of SHA-1. That's a big change you ought to tell developers about when it's their software that had to deal with those certificates.

Why not? When you're running a service like this, thinking instead of checking is not a wise move.

How much time would it have taken to whip up a script that just tests all of the apps in the store? Even if it's a week of an engineer's time, that seems worthwhile. Testing whether the apps in the store actually run when presented with a valid receipt should already be automated anyway.

I'm looking through Apple's guidance here:

https://developer.apple.com/library/mac/releasenotes/General...

The only thing they say about which version of OpenSSL to use is that you need to bring your own and not rely on the one that ships with the OS, because Apple is still shipping a version of OpenSSL from 2005. They do not say anything about which version of OpenSSL you need to use or what capabilities it needs to have.

I bet a lot of people were using Apple's OpenSSL for this originally, and when it was deprecated the path of least resistance would have been to bundle the same version Apple ships.

To be clear Apple's OpenSSL (currently 0.9.8zg) supports SHA2 despite its deprecated status. You can't link against it in El Capitan any more since they stopped shipping the development headers, but getting an OpenSSL that doesn't support SHA2 is actually somewhat challenging (You'd either need an OpenSSL compiled with OPENSSL_NO_SHA or OPENSSL_NO_SHA256 or one that predates 0.9.7h).

None of that excuses Apple not checking this of course. Security.framework can do all this validation -- did they not historically provide code samples on how to do this without linking an external library?

Sorry, I didn't mean to imply that their ancient OpenSSL doesn't do SHA-2, although I had no idea either way. They just say you can't use it because it's deprecated, and therefore off limits to App Store apps.

Historically, I believe the code samples were more or like the ones in the link I posted. I don't believe they ever posted anything that showed how to use Security.framework for the heavy lifting. I think they wanted to get everybody to bring their own code to make it so you couldn't pirate every app with a patch to one shared library.

App Store was introduced with Snow Leopard, when it was not depreciated.

Interesting, they use a version of OpenSSL that will go EOS at the end of the year? [1] It wouldn't support TLS 1.1 or 1.2, either, unless they've hacked that into their version.

[1] https://www.openssl.org/policies/releasestrat.html

It's not used, exactly. It's more of a historical accident.

Way back when, Apple shipped OpenSSL libraries as part of the OS and included them in the SDK. Naturally, some apps were built that used them.

Then Apple discovered that they couldn't really upgrade the OpenSSL they shipped with the OS, because OpenSSL doesn't maintain binary compatibility. They could ship a new version, but they were stuck shipping the old version as well, and the new version would suffer the same fate.

So instead, they reimplemented what they saw as the important functionality, shipped that as CommonCrypto and Security.framework, and deprecated OpenSSL to try to get people to stop using it. But they still ship it with the OS, because they don't want to break whatever apps do still use it. And they can't upgrade it, because that would defeat the whole purpose of shipping it.

Even today Mac OS X still ships with a copy of OpenSSL 0.9.7 for that reason. I wonder when exactly it was last updated.

No, it does not ship with 0.9.7. It ships with 0.9.8zg, the absolute latest in the 0.9.8 series. It is unfortunate that 0.9.8 is shipping at all (as it has numerous issues), but Apple has removed the development headers in El Capitan so nothing "new" can link against it now (although apps compiled against it on 10.10 and earlier can still use it). It will be interesting to see what Apple does when 0.9.8 is officially EOL at the end of this year. They may choose to backport fixes (if necessary) or they may just ignore it until 10.12 comes out and they decide to fully remove it.

I am talking about the multiple versions shipped for compatibility with old apps. It is not easy to access, but look for /usr/lib/libssl.0.9.7.dylib I think. I think it is used for apps compiled using Leopard or older SDKs, which I hope no App Store apps are using, but... BTW, back in the PowerPC days they even used to ship with the even older 0.9.6 (only got rid of it when Rosetta was killed).

Ah, yes, those are indeed shipped for ABI compatibility with old apps. Chances are that was last updated the last time 0.9.7 got a release, which would be Feb 2007 I believe (0.9.7m).

I think Apple later patched it to for example disable renegotiation I think.

You'd be right if the developers were distributing their apps on their own - once Apple has put up a service in the form of the App Store and are taking a 30% cut out of every sale, it's their responsibility to keep it running. If that means detecting that many apps use incompatible OpenSSL and communicating with the developers to address that issue before rolling out SHA-2 cert, so be it - all competent services companies do things like these all the time.

It's not as if there wasn't a way for Apple to detect this (if there wasn't then again it's their own fault) and it wasn't as if they couldn't have renewed with another SHA-1 cert for a few months until they figured out how to roll out SHA-2 one without causing a lot of people a lot of trouble.

> It's not as if there wasn't a way for Apple to detect this

Assuming developers were statically linking the old OpenSSH versions, how would Apple detect this? Are developers forced to reveal source code before submitting to Apple? Does OSX executables have some format to specify what libraries it's using?

I don't develop for OSX so I'm genuinely curious.

Apple is able to scan iOS code for e.g. use of nonpublic API's; certainly they can grep a binary for a recognizable chunk of OpenSSL code, too.

(Neither the iOS scans nor the OpenSSL scan will catch everything, of course.)

A recognizable chunk of OpenSSL code? It could be any statically linked and custom-compiled version, that's millions of possible versions, once you take into account compile options, optimisation levels and linking techniques. Good luck grepping for it! Even if the version number is in there somewhere, there's probably all kinds of false positives that you would hit. Plus, it could be due to other SSL libraries, not just OpenSSL.

In short, there's no realistic way to do what you are asking.

I disagree. Finding recognizable chunks of code is what antivirus software does every day. And while they're not perfect, the OpenSSL code in question here isn't exactly trying to hide, unlike viruses in real life. This was obviously an oversight, but I would expect that after this, they run occasional tests of the app verification process across all apps, and not just verify the apps themselves don't crash.

In fact, they must be doing this kind of thing right now -- how else could they contact all the affected developers?

I'm not terribly familiar, but these guys appear to be achieving some pretty sophisticated introspection into binary library usage:

https://sourcedna.com/

It would be interesting to see this applied to non-mobile apps.

Not what the parent said, true, but you can test for this easily: launch the app, see if it terminates with exit code 173 (receipt check fail) in the first few seconds if fed SHA-2 receipt but not the old SHA-1 one. Won't catch all (apps may validate later during runtime), will catch enough to give you some idea about the scope.

...by testing the apps still run?

For software of any complexity, and web apps specifically - you don't just deploy changes in Production - you'd need to have some type of test environment - staging, pre-production whatever you wanna call it. The test env is typically a replica of Production env - you deploy your change in test and point your clients to it. Sounds complicated but once you have the env setup done and replication processes defined it's dead simple.

For a company the size of Apple (with their exhaustive resources and long history), a comprehensive preproduction test that tests the action of upgrading a lot of the software... that should be within their capabilities. As in "don't just approximate what the users will be doing, actually do what the users will be doing".

Apple, third party friendly? Cold day in hell...

By comparison, Salesforce within this last year disabled a bunch of cipher options on it's HTTPS connections, and communicated it to their developers pretty well, even though most shouldn't/wouldn't have been affected except for edge cases...

Compared to this mess with a much larger impact.

I wish we could find a way to fix this "old SSL library/configuration" category of bugs across the board.

Neither OS distributors, nor app makers, nor sysadmins seem to be uniformly good at (or interested in) staying on top of this, and it means users get unreliable security.

Reply on news.ycombinator.com