Apple App Store STATUS_CODE_ERROR causes service problems | Hacker News Reader

Apple App Store STATUS_CODE_ERROR causes service problems

discussions.apple.com

287 pointswolfhumble136 comments

I can't commit code at CloudFlare because we use two-factor auth for the VPN (and everything else) and non-Apple apps on my iPhone are asking for my iTunes password. Tried airplane mode and apps simply don't load at all!

Total app-ocalypse.

Why do apps need me to be authenticated against iTunes to work at all?

It's a "nightmare scenario" downside of the iOS software model. When the device incorrectly decides that you are not authorized to run some binary, there's no way around that.

Makes sense. And this happened to me almost certainly because I upgraded to iOS 8.2 and my phone promptly crashed... hard.

Curious, which apps seem to be affected? Anecdotally, I hadn't seen any issues on my phone this morning with anything asking for a password, but on my laptop I did see a login prompt when i woke it from sleep.

All non-Apple apps. So far have tested Twitter, Hipchat, BBC Weather, Wickr, Authy, Google Authenticator, YouTube and Citymapper.

weird, I can open Google Authenticator (the only app we have in common from that list) just fine

You probably have a signature check cached that's still valid.

mine seem fine as well. I have a handful in common with that list. fingers crossed that they don't have to re-auth

While this comes too late to help you right now, I recommend looking at running a 2FA app on your laptop or desktop like this - https://github.com/gbraad/gauth so that you are not in this situation again.

Either that or grab a cheap Android handset and use it as a backup. The standard 2FA app on Android needs nothing more than occasional network connectivity to keep the clock in sync. You don't even need a Google account, the app is on FDroid.

Cloudflare is huge and many of us rely on it, so I hope you can easily avoid this predicament in the future - good luck!

>I recommend looking at running a 2FA app on your laptop or desktop

I very strongly recommend against doing this: If you do that, you are giving up a lot of security provided by that second factor as the malware you are using 2FA to protect against now also has access to the keys used to create the 2FA token.

This is a fair point of course, but running it on a second laptop is probably more secure than running it as a mobile app. You wouldn't run it on the same machine you are pushing production code out from, it could be a personal laptop with no access to company systems. I didn't make this point clear in my original comment though.

If you have malware, it can also act as a proxy requesting your codes and forwarding them (e.g. to disable 2FA). 2FA protects against password theft.

If your machine is compromised, it's over.

You're assuming the laptop has malware installed capable of pretty unrestricted access. At that point, all bets are off.

It can just forward code, relay cookies, etc. 2FA protects against someone peeking at your keyboard, or reused passwords, not malware.

If you're really desperate: When I last checked, Google Authenticator's keychain entries were not marked "this device only", so they can be extracted from an encrypted backup using something like "iphone-dataprotection" tools.

the app is on FDroid

I recently learned that there is an authenticator in f-droid, but not the authenticator, if one reads the notes at the top of the f-droid listing: https://f-droid.org/repository/browse/?fdid=com.google.andro...

I don't even know what they would want to stick in the Play store's authenticator above the already open sourced functionality.

"DISCLAIMER: This open source project allows you to download the code that powered version 2.21 of the application. Subsequent versions contain Google-specific workflows that are not part of the project."[0] The Play Store version is 2.49[1], but I also don't know what "Google-specific workflows" really entails.

[0] https://github.com/google/google-authenticator-android/wiki [1] https://play.google.com/store/apps/details?id=com.google.and...

Some messages in the thread mentions: - iTunes Store - Mac App Store - Apple Support - Game Center (https://discussions.apple.com/thread/6868907?start=45&tstart...)

Of these I have only tested "Mac App Store" and "Apple Support" and they don't seem to work.

I could log into the Apple Store though.

Apple TV is definitely affected as is iTunes Connect (at least for me.)

A lesson, perhaps, in why it's a bad idea to lock yourself (and your customers paying you hundreds to thousands of dollars every month) into third-party services (say, Authy) in order to get basic security features for which widely-adopted, standard alternatives exist.

Do you have an axe to grind about Authy? Because I can't get into Google Authenticator either so this is an Apple problem not an Authy problem.

It's about being tied to a third-party service. Currently, the one causing you a problem is Apple's. It could just as well be Authy's.

With standard TOTP, you could pull out a paper backup and use it to make any TOTP app on any platform work. But with Authy, you're screwed if they're down.

And it sounds like you're screwed right now because Apple is down. Again, this wouldn't be an issue if you were using standard TOTP.

Meanwhile, if I'd given in and setup 2FA on my company's CloudFlare account, I wouldn't be able to access it right now.

So yes, I have an axe to grind -- with CloudFlare. We pay you $200/month, and the only reason we can still access our account is because I've refused to setup 2FA on it with Authy.

Meanwhile, if I'd given in and setup 2FA on my company's CloudFlare account, I wouldn't be able to access it right now.

That's not the case. I'm in that position because of the fact that I got my phone into a bad state and can't reauth to iTunes. Others in the office have no problem at all and by talking to the technical support guys I know that I'm in an unusual position otherwise our customers would be very upset.

So yes, I have an axe to grind -- with CloudFlare. We pay you $200/month, and the only reason we can still access our account is because I've refused to setup 2FA on it with Authy.

What exactly is wrong with Authy that makes it completely unacceptable for you to use?

No Windows Phone client for a start.

I recently complained about the lack of a Windows Phone client on their Facebook page, and their response was "The service you've requested is currently not on our roadmap".

I had asked the founder about a Windows Phone client a couple of weeks ago and at the time he had said a WP client was 'very likely'. Not sure which is more recent, but I would also like to see an Authy client for Windows Phone.

https://news.ycombinator.com/item?id=9100644

What exactly is wrong with Authy that makes it completely unacceptable for you to use?

  Let's turn this device into a secure token
  Enter your Authy cellphone
  __+code__ __Authy cellphone number__

They're my private tokens. I don't want to set up an account with Authy Inc and I certainly don't want my tokens in your cloud. Sure maybe it'd be nice to sync across my devices, but not if it looks like it means doing so via somebody else's servers!

You realise that you encrypt the tokens you send and you can't restore them without that password, right? And that backups are opt-in only and you can leave that option disabled?

Do you not trust them to actually encrypt the data before backup? Or is there another issue?

Authy necessarily has the keys on its servers in cleartext. When you integrate them into your application, you send the code the user inputs to Authy's servers for verification.

Authy is a third-party authentication provider, it is not simply a synchronization service.

No, I don't realise any of that, because Let's turn this device into a secure token is the sum total of the information you get when you fire up the Authy app. No links to any explanations of why they want you to have an account, and I didn't care enough to go looking further myself.

But the real issue for me is: given a choice between (1) having my private tokens physically only on my own devices; (2) having an account and apparently some form of my tokens at a third party in another country susceptible to bulk espionage and subpoenas... why on earth would I choose (2) in today's climate?

Or in summary: I guess I don't trust them to actually encrypt anything in the face of legal threats.

> What exactly is wrong with Authy that makes it completely unacceptable for you to use?

This exact scenario is sufficient on its own. Also individually sufficient are the unnecessary revelation of personal information to Authy, and an aversion to perpetuating an authentication method that unsophisticated users can easily confuse with more secure methods that don't rely on third parties.

SMS and voice call codes should still work.

you do realise that Authy uses standard TOTP? And Google Auth? This is like complaining about Firefox being a "third party app" and if you were just a normal person you would parse the HTML in your brain directly from the output of your ethernet cable or something.

None of these tools have any form of lock-in, but Apple imposes its own layer, in this case.

Authy uses a modified form of TOTP. CloudFlare requires Authy, not a standard TOTP implementation. They make this very clear when you try to activate 2FA.

Authy works as a third-party authentication provider. Their servers are in the loop on every login. They aren't just a TOTP app + synchronization, they actually do the code validation themselves.

The correct analogy would be if every time you went to a website in Firefox, it asked Mozilla's servers if it was OK to go there. Also if, when installing Firefox, it demanded you create an account and give your cell phone number to Mozilla. (Or better yet, Google.)

Tbh, to setup sync or install extensions in Chrome, you need a Google Account, for which you need to give them your cellphone number. And it uses a blacklist hosted by Google to check if a page should show a malware warning, or not even display at all.

So your scenario isn’t even that far off.

I just installed AdBlock in Chrome, via the Chrome web store, on a Windows 7 box, that has never had a Google account logged in on it. So at the moment, I see no evidence that you need one to install extensions. Even if you did need one to install from the Chrome Web Store, you can also side-load.

With regard to the others, there are crucial distinctions:

* Chrome synchronization is entirely optional. You can use Chrome without ever logging into Google's servers. That's not the case with Authy.

* The malware blocklist feature uses data stored on your local machine that is frequently updated from Google's servers. It does not send the URLs to Google. Even aside from privacy implications, that would be annoyingly slow. And the block can be bypassed (last time I saw it, anyway) with a single click.

Please mail me at rdl at cloudflare.com to talk about TOTP/Authy/etc. I'd appreciate your specific concerns (and I have some questions).

If it affects every single non-Apple app then there are no alternatives out there.

That's very unusual — I've never had a non-Apple app ask for my iTunes password.

It's because the device has tried to get authorization for running the app (a unique, per UDID signature) and has made the assumption that for whatever reason your access has been revoked. That will normally never happen, it's just an odd side effect of whatever this downtime is. People not seeing this just happen to have hit a local cache of some sort and will eventually get it the longer the downtime goes.

Why do apps need me to be authenticated against iTunes to work at all?

I guarantee that someone would be screaming at Apple for not authenticating against iTunes under circumstance X, Y or Z.

I'm not sure whether it's better to be inconvenienced for a day, or for Apple to attempt to cover your back if something looks amiss in the iPhone's systems.

(which, from the sounds of your upgrade and hard crash, this may well be - or "simply" some form of empty cache)

It feels a bit like six of one, and half a dozen of the other, the painful bit being that ideology doesn't help you get your work done today!

I wouldn't scream at Apple if they backed off from protecting/micromanaging/dictating every aspect of the using their products.

What if you were an app developer who had a paid app on the store? Would you want Apple to sit back while people pirated your app? Would you want to have to implement copy protection yourself? Do you think every app should do that?

iTunes authentication is just a DRM thing, making sure you've paid for apps before you can run them. Security is handled by code signing, and more importantly by sandboxing. The only people who would scream at Apple for not authenticating against iTunes are those who don't understand this, or those who think Apple needs to enforce strong DRM for third-party apps.

You should investigate DuoSecurity, perhaps tied with some Yubikeys - they scale TSV to enterprise scale and have systems for handling scenarios like lost phones.

May be a coincidence, but Duo on my iPhone is having issues. The request to authorize shows up, but then I get 'Unknown Error - The request timed out.'

Using the code does work, so I don't think it's Duo.

This is happening cause apps validate against the app store to ensure you haven't jailbroken them.

> Tried airplane mode and apps simply don't load at all!

Does this mean you normally can't ever use 3rd-party apps in airplane mode? Because that is definitely not the case for me.

The apps authorization to run has been revoked. So now it has to connect to whatever is down to be re-authorized. That is the problem.

I see. So if it tries to run and gets revoked, you're screwed until Apple comes back up. But in theory, if you had been in airplane mode the whole time this incident has been going on, you'd be fine?

Exactly.

same case happened to me once when I'm on a 13hr flight in Jan. Simply all non-stock apps doesn't work anymore, and back to normal when I landed and connected to Internet.

I suppose this is a good example of where Azure MFA can be useful. Rather than having an app generate a code, it can be configured to call your phone instead. I hadn't really cared for this option, but as long as your phone can receive calls, you can still authenticate, so it protects against this sort of situation.

CloudFlare had a huge DNS outage yesterday that broke half the Internet for NY/NJ and some other location.

Hopefully these lessons teach us that single points of failure are bad.

I'm always puzzled whenever I have to enable a Cloudflare script on a site that clearly doesn't have enough volume to need it and would have been more robust and simpler with a traditional data center. It also annoys me to no end that I have to permit JavaScript from random hosts to support a cloud server that should be completely transparent to me.

The site just probably doesn't like getting DDoSed.

Is that true? Did CF break Verizon or was it the other way around?

Authy chrome browser app.

Have somebody else pull with an Android pull from your local repo and push to CloudFares system.

It is an interesting variation of the Airplane situation, but no connection is no connection.

No mention of this on https://www.apple.com/support/systemstatus/ though . . .

[Update 15:02 11-03-2015, GMT+1]

* iTunes Store - All users are affected

* iCloud Account & Sign In - All users were affected

* iCloud Mail - All users were affected

This is what I think is the bigger news and deserves far bigger pressure than the downtime. Server downtime can always occur, but then you need to inform your paying customers (such as developers whose apps are counting on your services!) what the issue is. If you are paranoid and don't want to reveal information (for example if it is due to a hack of some kind), you AT LEAST should say something along the lines of "we know there are issues, we are on it."...

But it is absolutely unacceptable and inexcusable to have hours of downtime and have BOTH your end-user and developer status page still be completely green across the board.

Infuriating!

Tried to contact Apple via: https://www.apple.com/support/contact/ (As mentioned on the bottom of the page: "If you are experiencing an issue not listed here, contact support"), but that does not seem to work.

I keep hitting this error http://i.imgur.com/D6TWBuF.png

This just how Apple has always been, not taking care of what any sides want, until it brings money.

This concur with the #watchFiasco of apple, explicitly saying "you're pigeons, and we deliver"

I'm not sure if it's as simple as that, certainly OSX has been poorly neglected with 10.10 but generally their cloud services are rock solid (at least in Australia) where Microsofts alternatives suffer from weekly unexplained outages and poor performance on a global level. The number of Amazon instances that crash and fail to ever come back or new instances that start broken is also often dismissed it seems.

> generally their cloud services are rock solid

Rubbish. I just sold the kids' MacBooks and iPads because iCloud and Pages/Numbers is a piece of shit from a reliability perspective. Add to that the WiFi stopped working reliably with 10.10.

iMessage still suffers from massive consistency failures; due to the incoming messages not being chronologically ordered even small network glitches mean things get confusing fast. iCloud at least in part seems to run on Microsoft Azure, amusingly enough.

I'm not questioning your anecdata, but (to offer some of my own) I have never experienced this and none of the people I work with or socialise with have ever brought it up. I only hear people on the internet talking about it as if it's true of the service as a whole. It definitely isn't.

Yeah I've never had problems with it either?

Couldn't agree more, transparency and admitting fault or failure quickly is key to trust with clients.

Perhaps coincidentally I received an unknown charge from iTunes at 1:13am. Since I do not remember making this purchase I would really like to see what was purchased. Unfortunately the store is down.

Maybe they updated it, I'm seeing the page list mail and auth. outages from ~4-~8 AM.

I assume this is CDT. I'm seeing ~5-~9 AM EDT.

Actually it's not showing outages. They're showing them as "service issues" (yellow), not "outages" (red). Same as Amazon when their EBS or whatever is on fire they show it as a green check with a small blue "(i)" instead of a proper red "it's down".

Why is so hard for service providers to admit that things are broken? Is it an attempt to weasel out of SLA uptime promises?

edit: Apple just updated the page to show nice big red outages. Kudos! Hopefully they have learned from whatever kept the status page from updating and it keeps working going forward.

It’s weird seeing specific services slowly get the red flag on the status page that have seemingly all gone down at the same time.

I can’t listen to any of my music from iTunes Match, which still shows Green on the status page. But I get an error about the iTunes Store not being able to process purchases (???) when I try to listen to my music, where iTunes Store does show a red flag. So the status page is telling a half-truth. What a mess.

It seems that the "Detailed Timeline" at the bottom shows an outage for "iCloud Account & Sign In" earlier today. That is probably the issue for the original post.

This is just another example of Apple's severe structural deficit on their cloud engineering teams. From a user perspective, Apple's backend engineering is in desperate need of a management shake-up.

I don't know about that — I've never personally encountered an issue with Apple's cloud services, despite using them for years. Hell, I'm not even experiencing the problem that apparently hundreds of other people are.

Are there other examples of Apple's cloud failures?

I'm having problems every week with Ulysses' iCloud sync: syncs that won't happen, lost data... I back up my data by hand to text files just to be able to recover. It's insane how much it fails and how little visibility over what's going on it offers.

It's interesting, because I only use iCloud sync heavily with two apps, but I never have issues with with either of them, which leads me to suspect it's partly a development issue. I hesitate to blame the app developer per se though, because it's also Apple's responsibility to ensure that iCloud integration is easy to set up and difficult to get wrong.

iCloud sync (the pre-iCloud-drive incarnation) has been fraught with issues. A lot of folks in the iOS dev community have been griping (and writing really great blog posts) about this for some time now. There are times when iCloud just goes off the rails. For quite a while I'd been one of the "lucky" ones, but since then I've had to jump through workaround hoops to obliterate all of an app's iCloud data to fix syncing errors, seen things like two Macs on the same Apple ID that couldn't see each other's files and so on.

Imagine the furor if you could log into the same account from Dropbox in two different places and not see all of the same files.

AFAICT, the iCloud Drive revisions were essentially an acknowledgement that the first iCloud backend API and implementation was just a great big mess. Seems to be a huge case of overpromise and underdeliver, which sadly seems to be the banner motto of Apple's cloud offerings.

> This is just another example of Apple's severe structural deficit on their cloud engineering teams.

Apple rarely has infrastructure issues this large. Mind detailing your "other examples" for evidence?

iCloud is widely considered to be troublesome by many different developers.

I'm wondering if Apple will, at least, issue iOS developers with an extra day on their membership.

More notable is how many apps, and thus businesses and individual developers, this is affecting in terms of lost sales.

I doubt Apple will compensate in that case though.

It looks that they are losing $2,054,794 per hour of downtime [1].

[1] http://www.dailymail.co.uk/sciencetech/article-2989656/Apple...

Over 3M USD then?

I noticed this issue early this morning (late last night), about 0130 EDT, so as of now, 10.5 hours...

I am willing to wager that the average sales is not uniform over the day/week. Now we are getting towards a full day, but I would hardly count the hours from 0130 to 0500 as downtime for lost sales.

Unless you live in another time zone. In Europe it has been nearly all waking hours (unless you work in a night shift).

That assumes that failed sales during an outage are lost forever, not merely delayed.

I've heard from various third-party developers that store downtimes (of reasonable lengths) tend to be followed by corresponding sales spikes, so that ultimately nothing is lost. I have no idea if that applies here, but it's reasonable to think it would at least partially.

I'd say that a sizeable part of App Store sales or downloads are impulsive purchases: i.e. you think you need something or are in the mood of playing some game and you can't resist hitting the button. If you can't make that impulsive purchase now, you tend to forget about it until you are in the right mood again. My take is that a very high amount of sales has been lost, not deferred.

CNBC are reporting Apple made the following statement:

“We apologize to our customers experiencing problems with iTunes and other services this morning. The cause was an internal DNS error at Apple. We’re working to make all of the services available to customers as soon as possible, and we thank everyone for their patience.”

Seems like DDoS toward the US?

http://map.ipviking.com/

Here's something that I've always wondered: Where do they get this data?

In this case, at least, honeypots. From the linked page:

> Every second, Norse collects and analyzes live threat intelligence from darknets in hundreds of locations in over 40 countries. The attacks shown are based on a small subset of live flows against the Norse honeypot infrastructure, representing actual worldwide cyber attacks by bad actors.

Probably backscatter?

It's hard to judge whether what is on this page indicates anything without some reference or historical data.

I think this is as good an illustration as any about the benefits of a decentralised ecosystem.

Siri isn't very helpful this morning either. I don't think she can talk to the mothership.

they updated the status page http://www.apple.com/support/systemstatus/

"iTunes Store Service is unavailable for all users." Wow, how much will that cost to Apple!

Considerably less than ($revenue_per_minute * $downtime) would have you believe because few people are going to attempt to buy an album, have their purchase fail, then never attempt to repurchase it. True, some number will use another online seller, but I suspect they're in a tiny minority.

My Apple TV has just randomly rebooted. Wondering if that has anything to do with this...

My Roku box wouldn't connect to Netflix this morning either. I wonder if there is something else going on here. :/ Maybe there is a problem with AWS or the internets in general.

steampowered.com is also down.

So like a normal Tuesday night for me and my Apple TV.

My wife reboots it manually before watching anything, just in case.

I can hardly fathom how many sales this downtime is costing Apple and developers.

The entire business model of IAAS depends on the idea that the infrastructure is 24/7 accessible. The loss of sales during this time is nothing compared to the loss of confidence in the company will have. Measuring the damage will be an interesting discussion.

Everything seems fine here in Melbourne, Australia. I am disappointed that their status page doesn't reflect the issues that it seems many are facing, I struggle with this a lot with Microsofts cloud offerings (o365 etc...) which go down or are partially unavailable every week - Yes you heard that right - weekly.

The model of trust with vendors, large and small alike has its downsides and the widespread impact of centrally authenticated systems is one of them. Is there an answer? I don't know but it certainly comes down to convenience vs trust in many cases.

I'm also in Melbourne at the moment (on BigPond Cable), and I can't even download Xcode 6.2 because of, you know, STATUS_CODE_ERROR.

See articles in Mac Rumors (http://www.macrumors.com/2015/03/11/app-store-itunes-connect...) and The Next Web (http://thenextweb.com/apple/2015/03/11/the-apple-app-store-i...).

Glad I saw this—having problem restoring an iPad and I could not for the life of me figure out what was going on. Now, this iPad has to leave the country in about 30 minutes...

I was getting this error when trying to update my apps through the App Store. After entering in my password it shows in red letters under the Apple ID entry field "STATUS_CODE_ERROR". That was a couple hours ago.

I just tried again and it said, in red letters, in the same spot, "plist parsing error". Clicking "sign in" returns these errors apparently interchangably.

Apple Multiple Store Services Working Again:

https://www.apple.com/support/systemstatus/

http://i.imgur.com/O7lDwx4.png

iTunes Connect is down as well. I've been unable to login.

Also https://news.ycombinator.com/item?id=9186162.

iTunes and App Store are both unable to connect across any of my devices this morning here in the US. Very strange.

AppleTV can't rent films :-(

iCloud is down as well. I am not able to edit my files. :(

my macbook has come up with with this STATUS_CODE_ERROR when trying to log in to App Store... when will it be back to normal or how can i fix it???

Why today app store is not working??

Why app store is not working today? I am unable to download the apps from app store.

We should seek some news from Akamaï

i couldnot download and update apps in iphone.

monoculture ....

i cannot update and download new apps. wht should i need to do solve it?

Welcome to The Cloud

You're just holding it wrong.

Computers, how do they work?

perhaps those west coast guys arent awake yet? too busy recovering from their launch parties.

the ONE day I want to buy Frozen from the iTunes store...

Thanks apple, for dashing the hopes and dreams of this twenty-something, now bawling their eyes out and screaming for Frozen.

download torrent now, buy later (ideally voting with your wallet by choosing another seller)

HEADS UP TO IOS DEVS: You may want to update your iPhone to iOS 8.2. You'll need to download the new XCode to deploy the app after that but it's not possible at the moment due to STATUS_CODE_ERROR.

Reply on news.ycombinator.com