Mac Store Apps Stopped Working Due to Expired Security Certificate
techcrunch.com
techcrunch.com
"Apple issued a new, stronger (SHA-2) Mac App Store certificate in September, before the older (SHA-1) one expired, as planned. The new Mac App Store certificate was using the current, strong SHA-2 algorithm. However, some apps were running receipt validation code using very old versions of OpenSSL that don’t support SHA-2.
OpenSSL started supporting SHA-2 in 2005, which is why Apple didn’t foresee this issue."
How much time would it have taken to whip up a script that just tests all of the apps in the store? Even if it's a week of an engineer's time, that seems worthwhile. Testing whether the apps in the store actually run when presented with a valid receipt should already be automated anyway.
I'm looking through Apple's guidance here:
https://developer.apple.com/library/mac/releasenotes/General...
The only thing they say about which version of OpenSSL to use is that you need to bring your own and not rely on the one that ships with the OS, because Apple is still shipping a version of OpenSSL from 2005. They do not say anything about which version of OpenSSL you need to use or what capabilities it needs to have.
I bet a lot of people were using Apple's OpenSSL for this originally, and when it was deprecated the path of least resistance would have been to bundle the same version Apple ships.
None of that excuses Apple not checking this of course. Security.framework can do all this validation -- did they not historically provide code samples on how to do this without linking an external library?
Historically, I believe the code samples were more or like the ones in the link I posted. I don't believe they ever posted anything that showed how to use Security.framework for the heavy lifting. I think they wanted to get everybody to bring their own code to make it so you couldn't pirate every app with a patch to one shared library.
Way back when, Apple shipped OpenSSL libraries as part of the OS and included them in the SDK. Naturally, some apps were built that used them.
Then Apple discovered that they couldn't really upgrade the OpenSSL they shipped with the OS, because OpenSSL doesn't maintain binary compatibility. They could ship a new version, but they were stuck shipping the old version as well, and the new version would suffer the same fate.
So instead, they reimplemented what they saw as the important functionality, shipped that as CommonCrypto and Security.framework, and deprecated OpenSSL to try to get people to stop using it. But they still ship it with the OS, because they don't want to break whatever apps do still use it. And they can't upgrade it, because that would defeat the whole purpose of shipping it.
It's not as if there wasn't a way for Apple to detect this (if there wasn't then again it's their own fault) and it wasn't as if they couldn't have renewed with another SHA-1 cert for a few months until they figured out how to roll out SHA-2 one without causing a lot of people a lot of trouble.
Assuming developers were statically linking the old OpenSSH versions, how would Apple detect this? Are developers forced to reveal source code before submitting to Apple? Does OSX executables have some format to specify what libraries it's using?
I don't develop for OSX so I'm genuinely curious.
(Neither the iOS scans nor the OpenSSL scan will catch everything, of course.)
In short, there's no realistic way to do what you are asking.
In fact, they must be doing this kind of thing right now -- how else could they contact all the affected developers?
It would be interesting to see this applied to non-mobile apps.
Compared to this mess with a much larger impact.
Neither OS distributors, nor app makers, nor sysadmins seem to be uniformly good at (or interested in) staying on top of this, and it means users get unreliable security.
One of my Macs is stuck on Snow Leopard. On Lion & El Capitan, I sign in to the App Store again and it's resolved. But on Snow Leopard, it doesn't display the sign-in error like 10.7 - 10.11 do. It just complains and tells you to redownload the apps from the App Store.
But I can't. Apple doesn't let you download old versions from the Mac App Store. Byword 1.5.2 is the last version that works on 10.6, the current 2.x version requires 10.8. Similarly with Tweetbot, Mémoires, Space Gremlin, etc. Unless Apple magically allows old downloads, I don't think this can be fixed without Apple patching Snow Leopard. I'm not going to hold my breath.
It really seems like Apple has permanently broken my purchased apps.
(Thank you for trying to help, though!)
Actually the other reason for me also was Visual Studio - I develop quite a bit of C/C++ code and recently moved to developing an Android app - VS Emulator for Android is a godsend - you can't do those things on OS X without trouble - Xcode is buggy and the emulator with HAXM is crashy although that's Intel's fault.
And bonus points to Dmitry of Coding Robots <https://www.codingrobots.com> for offering this, because this was partly my own fault. I'd moved and archived the app, but not actually deleted it, and this didn't clear the App Store install receipt from my computer. I had to actually move the app to the trash for OS X to remove the install receipt. Once I did, the App Store let me reinstall Mémoires again, now code signed until Feb 7 2023. But I'll now be using the non Mac App Store version Dmitry just sent me.
[No such luck with Byword though, as I still can't download the old Snow Leopard version of the app from the App Store.]
For instance, they could have:
- Quietly noted the issue in a file somewhere, waited 24 hours, and checked again.
- If the problem was still present after 24 hours, they could have made the software discreetly submit a problem report to Apple (but still launch and work correctly).
- After 48 hours, they could issue a background notification to the user indicating that a problem has been detected in the program and to “please check for updates on the App Store”; but again, continue to work correctly.
Or in other words, there were only about a half dozen more reasonable things they could have done that didn't involve an Amazonian break-things-immediately type of response.
If my driver's license expires and I drive, I'm breaking the law. Sure, a judge or cop could be lenient, but there is no legal requirement (or justification?) for them to do so.
IMO, tossing up a quick warning for the next month after expiration seems like a reasonable solution. Sure, something failed, but expired certificates are almost never a sign of a real problem.
How does this have anything to do with the issue? With a mechanical object, you have risks and probability of failure. The certificate or driver's license or credit card doesn't _fail_, it expires.
PS: With an expired license the number of days past expiration is important in most states. Wait long enough and you need to retake the driving test, so clearly before that point it's still somewhat valid.
You're confusing things. The validation fails, as in it doesn't meet a certain requirement, the certificate; it isn't damaged or otherwise broken. A mechanical object fails in that it takes physical damage. You're striving for some semantic technicality to save your argument and I just don't think it's important.
> An expired certificate is clearly a lower risk than a revoked certificate, so only having one failure mode regardless of cause is unnecessary.
Why? You're just throwing an assertion out there that is false and expecting me to believe it's true. Additionally, if there is a 2-day window where an invalid certificate is valid, then all that does is make the expiration date 2-days into the future. If the certificate's owner doesn't notice an invalid certificate until after it doesn't validate, then they won't notice until after the window anyway.
> PS: With an expired license the number of days past expiration is important in most states. Wait long enough and you need to retake the driving test, so clearly before that point it's still somewhat valid.
That doesn't mean an expired license is valid. That simply means there is an implicit expiration date for the validity of your driver's exam.
Your not limited to a single response. You could turn the URL red for a week and then after that do a popup etc. hell you could stup a browser to ignore expiration dates with minimal impact.
Remember, certificates are not some binary entity. You can have high and low trust certificates ex: trust to sign OS code, trust to sign user land code etc.
Your stuck in binary terms which are useless from a security standpoint.
And what does this buy you? Nothing, honestly. It just means that the expiration date is a week in the future from the date on the certificate. _You aren't actually changing the problem, just renaming it._ We could show red a week before the certificate is set to expire _now_; what does a grace period buy anyone?
Also, there are still have users that don't understand how to check if you have an SSLed connection, and you expect them to make a judgment on if an expired certificate is safe?
> hell you could setup a browser to ignore expiration dates with minimal impact.
No, no you couldn't. This means that if an old key, or one you've lost, is compromised, someone would be able to impersonate you, and you may be _unable_ to revoke a certificate. At least now, there is a definite amount of time that a scenario like that could happen for, you're proposing it happen indefinitely.
Moreover, it also becomes a forced upgrade requirement, viz sha2 certs currently. Sure, some longer-lived certificates may not have been updated (or supported currently), but they're not the rule. Most sites updated though a matter of course and never realized the security implications.
Additionally, the revocation status of expired certificates isn't currently published. Granted, this is a bureaucratic decision, it keeps the CRL size much smaller than it would be otherwise. (And again, it assumes you still have the keys to revoke an old certificate!)
> Your stuck in binary terms which are useless from a security standpoint.
Right, security isn't a binary thing. However, it is often based upon binary things, as they are easy to check, understand, and present. I also resent you calling me "stuck" when you're essentially asking for the exact same thing, just differently.
You can always revoke certificates.
what does a grace period buy anyone?
It avoids bad publicity and lost revenue. At some point you don't want to continue operating with old certificates, but breaking without warning is a terrable idea.
You have warning. You know when it will expire the day you buy it.
You still haven't told me how a grace period is different from extending the expiry.
A reasonable thing to do instead is to give a warning like "The security certificate for app X is expiring in two days. Please update it" or something like that.
OR don't have expiration dates on a cert, if that is preferable.
Being this strict at the cost of usability isn't about security, it's about DRM and making sure you don't get to keep using your software after the subscription has expired.
It's completely ridiculous and has me vowing never to buy (or get for free) anything from the Mac App Store again. Unfortunately I have little choice on iOS.
I also think you're perhaps extrapolating a bit too much from this incident. Did the MAS breakage really truly suck? Yes, it did. Was it really embarrassing for Apple? It probably was. Did it cause grief for a lot of people? Yes. But the big question is, does this mean we can't trust the MAS to not accidentally break all our apps again? No, I don't think this incident means that at all. I can't think of any other reason for all my MAS-installed apps to suddenly break besides an expired certificate, and Apple's now renewed the certificate. Worst-case, maybe it'll break again in a few years when that certificate expires, but I bet they're putting in place some procedure or mechanism to guarantee that this won't happen again.
All that said, there are certainly plenty of legitimate ongoing grievances with the MAS. I won't dispute any of that. But vowing to never again acquire any app from the MAS over this one incident seems somewhat absurd.
Before this incident happened, I thought that App Store apps which existed on your computer and had the proper credentials (verified by running the app successfully at least once) were permanently good to go. Verification happens offline, after all, so there's no room for error and no need to trust anybody.
Turns out, no, all of these apps come with a built-in expiration date, and you need Apple's cooperation to extend their life. They've fixed that. What else lurks in there? If they didn't actually intend to build a system which is once-and-done, then I really have no idea, and I can't trust that.
I don't see why it would be "absurd" to decide not to patronize a system which has demonstrated that it can't be trusted. I mean, what exactly am I losing by no longer buying there? Virtually all of the good apps are available outside of the store anyway. Used to be that I'd lean towards the App Store version because of convenience. Now I see that this is the wrong choice, so I won't do that anymore. So how is this "absurd"? Because it's not the choice you would make?
So really, you're just upset about my wording. That is, to borrow a term, absurd.
But please, continue telling me how my statement about my own actions is "absurd" based on your opinion of the situation.
Nah, can't be, it must be the fanboys who follow me around downvoting you.
――――――
I had a combination of things happen where I installed an upgrade, and it effectively locked me out of my files. I had to pay for a new Keynote version to get around the problem (I no longer remember all the details ... I think it had something to do with an iLife subscription vs. standalone and auto-upgrading my data file version).
I was REALLY pissed off. And vowed I would never buy software again where I didn't control the physical copy.
I have an issue with the concept of MAS as a monopoly (the same I've had with Windows Update), but MAS as a tool is undoubtedly a good thing.
I used the Mac App Store because it really is just very convenient. But I also thought it worked. Now I see that it doesn't, so I will not be obtaining software from it again.
Source: http://www.macrumors.com/2015/05/07/redacted-mac-app-profits...
They say after they left the App Store, their revenue went up 44%:
https://www.panic.com/blog/the-2014-panic-report/
"I was pretty nervous to be pulling Coda from the Mac App Store. But when we finally did it, I felt an incredible, almost indescribable sense of relief — mostly because as we began to wrap up bug fix releases, we were able to immediately post them to our customers within minutes of qualifying them. My god. That’s how it should be. There’s just no other way to put it — that’s how you treat your customers well, by reacting quickly and having total control over your destiny. To not be beholden to someone else to do our job feels just fantastic. (Also to not pay someone 30% in exchange for frequent stress is a fine deal.)"
The point is that 94 units with $452 in sales makes you eighth top paid app.
So, unless you are top 10, as a developer you can expect a maximum revenue in a year of $120,000. IF you can make that EVERY DAY ... and you can stay in the Top 10 ... and you can't. And Apple takes $40K of it.
And most people will never break the top 10.
I knew things were bad, but that's disgustingly bad.
The problem is no-one is going to pay that much money for an app that lets you add black boxes to an image. You have to deliver real value worth paying for. Think apps like Day One, Transmit, DaisyDisk, 1Password, Reeder.
Moreover, your numbers are off, you don't need to be anywhere near the top 10 to make over $120,000/year. In fact, you can be well outside the top 100 if your per-unit price is high enough.
Oh the other hand, if you do your own marketing and use the app store as a distribution channel, you could probably do just as well (or better) as if your app wasn't in the store. Depends on whether the benefits of the store are worth the 30% cut they're keeping.
What possible problem could you have with Windows Update? WU is Microsoft's platform for updating it's operating system, core software, and hardware drivers. It isn't a 3rd party app store and doesn't sell anything. In fact they'll host 3rd party drivers for Windows compatible hardware for free as long as the OEM or IHV goes through the proper certification process. It's not even a monopoly because you can disable it and use LanDesk or some other vendor's update platform.
Were you just trying to get a dig in at Microsoft because you don't like them?
Eventually they got around building an apt-get clone targeted at opensource developers, and an appstore for Metro apps nobody wants. In 15 years, that's not a great track record.
WU has never really been about software installation or upgrade, it's focus has always been about bug fixes and security patches. Silverlight and IE are the only apps that have ever been upgradable via WU and for IE that's been a relatively new occurrence primarily driven by getting users off of older insecure versions. So really Silverlight is only app ever distributed via WU.
You also have to consider that APT was a solution to the horrible experience of getting and installing Debian libraries and applications which still is a pain in the ass without APT. Windows never had the problem, apps that were available for download were easy to download and install.
Keep in mind that APT is for opensource software and doesn't have to deal with authentication and authorization of software licenses.
And since WU isn't a store front and most Windows software isn't free, opening up WU to 3rd party apps would probably have come with distribution costs that I would wager most app developers wouldn't pay.
So again not a very fair comparison.
Is the Mac App Store really not popular? One of my reasons for wanting to get away from Windows again is that I miss my central package management from linux, which is what I kind of thought the App Store did (besides collecting Apple's cut).
There's Chocolatey on Windows. The best I can say about it is that.. erm.. their hearts are in the right place.
I love Cask but I'll choose App Store apps over their counterpart until the app devs can improve licensing and settings syncing.
So as someone who didn't really know what was going on: It didn't really affect me too much, most apps seemed to still work partially (1Password, etc.). Still it didn't seem annoying because I thought it was the upgrade process.
I still love Apple hardware, but they keep screwing up the software side of things on both OSX and iOS.
Installing a different OS (like Windows 10, or Debian) can be a great way to give older hardware new life & continued productive use.
[1] Yes, Macs support mice with more than a single button now ;-)
“The last versions of OS 9 were pretty good about right mouse button support. Most mice should work fine.”
I would test it myself on my iMac G3 (Summer 2001), but unfortunately, I already upgraded it to Mac OS X v10.4.11 Tiger, and it’s getting active use, so I can’t exactly go downgrading it at the moment.
――――――
Hopefully they've actually fixed this now but I was pretty confused about what could be going on. The only other time I've gotten messages like that is when I'm opening an unsigned application Apple doesn't think is good, and the Security settings say to only allow trusted software.
Hopefully they're aware of what's causing this bug and will fix it in an OS update.
Of course, when the certificate is actually expired, and they haven't replaced it yet, then the "damaged" dialog is reasonable, because the system cannot get a valid replacement receipt. But once they replaced it, OS X should have just started working again. In fact, I'm going to go file a radar right now.
It involves analyzing full logs of all supported client enctypes and tracking down the full set of "flavors" of clients that only do SHA-1.
At the end of the day, you're going to break people, and it's all about minimizing how many people that is. Imagine the situation with hardware devices in the field from ten or so years ago. You can't update them and their software rev only supports SHA-1. What do you do?
(You break them.)
Isn't then the app store validated with the same method?
The expired certificate was part of the receipt that gets generated for an App Store app to indicate that you have the right to use that app on your computer. Stuff that ships with the system doesn't need this and so doesn't have receipts.
2035?
>The new Mac App Store certificate was using the current, strong SHA-2 algorithm. However, some apps were running receipt validation code using very old versions of OpenSSL that don’t support SHA-2.
I sure hope it's not a SHA-1 expiring in 2035...
You imply that Apple is going to keep using the cert until 2035. I can assure you they won't.
The nice thing about root certs is the private key is only needed to sign intermediate certs and CRLs. This means they can be kept offline in a secure location, and only accessed once every few months or so to sign new intermediate certs or CRLs. The actual crypto typically happens in a special locked-down piece of hardware, so that the actual private key never touches the memory or disk of the computer being used.
The whole process is called a "key ceremony" and follows strict procedures, with technical staff and outside auditors watching every step.
As far as I know, no CA has ever had its root key compromised. It would require physically breaking into a secure facility, or factoring of public key. Considering there's much lower-hanging fruit for anyone attempting to attack PKI, I'm not too concerned with a 20 year root cert lifetime. (Like you say, they could just remove it from future versions of OSX before then!)
[1] http://furbo.org/2013/10/21/mac-app-store-receipts-and-maver...
So, instead of buying overpriced software from the mac store, just building it from source.
“Apple issued a new, stronger (SHA-2) Mac App Store certificate in September, before the older (SHA-1) one expired, as planned. The new Mac App Store certificate was using the current, strong SHA-2 algorithm. However, some apps were running receipt validation code using very old versions of OpenSSL that don’t support SHA-2. OpenSSL started supporting SHA-2 in 2005, which is why Apple didn’t foresee this issue.”
Since I seriously doubt any of Apple apps would be using the older SHA-1 certificate, this probably only affected third-party apps.
――――――