Apple Dev Center outage | Hacker News Reader

Apple Dev Center outage

developer.apple.com

226 pointsgurgeous138 comments

The entire Apple dev center is down. That makes it impossible to download the iOS or Mac SDKs, including the iOS 7 beta. It's also not possible to provision new devices or update expired certificates. The developer forums are down too, which is somewhat frustrating since that's the only place we can discuss the still-under-NDA iOS 7.

iTunes connect appears to be up.

Apologies for those who posted this earlier. The headline gets more interesting as the outage continues.

I can confirm iTunes connect is up because our app entered review and was rejected within that last 3 hours :(

Sorry to hear that. I just had an app get approved and published an hour ago.

What did you make ?

No forum means no complaints which future developers can look back on...

The simpler explanation is that the cluster running all developer-related sites is down. It doesn't always have to be a conspiracy.

Oddly enough, the forums went down sometime after the dev center. I was able to access them yesterday, and there were no posts on the issue. At the time, I figured that a new beta might be going up or that it was a short outage.

Something real, real bad happened down Apple way. The only explanation that passes the sniff test for me is a security breach. A big one. Playstation store big. I mean, it's a huge outage in the middle of the week. Nobody does that if something hasn't gone disastrously wrong. And iffy as Apple's web infrastructure historically has been, it's hard to imagine a system malfunction that they couldn't route around in some kind of reduced capacity in this amount of time.

So. Is Apple gonna drop trou and own up to what the hell happened, or are they gonna get the servers back up and pretend that we saw all this smoke and no fire?

Well, apply the standard Apple social introversion: it's down, they're telling you it's down and will be for a while, it will be back up as soon as they can fix it, what exactly is wrong isn't your problem, when it's back up you'll know and it won't matter what happened so long as it gets up and stays up.

Well, the App Store is how I make my living, so their developer portal's uptime directly affects me. Especially if it was a security breach.

And the argument that it won't matter as long as it gets up and stays up has already been invalidated by this outage. If they were perfectly in control, this wouldn't have happened. So we know for a fact that they are not 100% in control of the website. (I mean, obviously, 100% control is a mathematical impossibility.) But how much less? What happened? If it was an hour, or even two, I'd let it ride as a cost of running a big-ass data center. But this... warrants an explanation.

>> "Well, the App Store is how I make my living, so their developer portal's uptime directly affects me."

How much is this outage affecting you? You can still provision devices etc. through Xcode, documentation's still online. I'm also a full-time iOS developer and hadn't even noticed it had gone down I go to it so infrequently.

I need (!) to document a sequence of steps thru a process in the Member Center. Can't do that if I can't get to the Member Center.

California state law requires companies to disclose security breaches. So, if such a thing did occur, you should expect to see one sometime within the next few days.

source: http://en.wikipedia.org/wiki/Security_breach_notification_la...

If you don't see such a notice, than it is (was) likely some other issue.

I've had an active developer program account since 2008. While various tools have gone offline now and then, and there are planned shutdowns around the holidays and during keynotes, this incident seems unique.

Wonder what's going on over there. This outage makes the provisioning portal inaccessible. That could mean very real work disruptions, since no new code signing credentials can be made.

NSA hooks in the dev forums? ;-) Crap, I'm not getting hell-banned am I?

Anyway, yeah. It's gone down a couple times for scheduled maintenance and we've been notified. This one wasn't mentioned at all. No surfin' the forums today/yesterday. boo.

I've managed to successfully provision new devices and run dev apps on them yesterday using xcode's organizer, so at least there's that.

Did you need to do anything special? Keeps timing out when I try to use it.

No. I had to add a few new iPads, so I opened the organizer, clicked the 'use for development' button and it was good. It took a little bit for the actual provisioning profile to update but I was able to run the dev app on the device right away.

This was yesterday morning. I haven't tried today, though.

My guess it's the SAN. Many enterprise deployments I've seen will expose the same binaries across the many servers in the cluster.

And it fits in well with an across the board fault.

There was an incidence of what I suppose we should term "unannounced maintenance" a few months ago.

Craig Hockenberry: "Just talked to WWDR on the phone: no ETA."

https://twitter.com/chockenberry/status/358315230742331393

Amusingly (or perhaps frustratingly) their status site is showing all-systems-go: http://www.apple.com/support/systemstatus/

None of those are within the Developer Center. It's very possible for those to be up and running while dev portal is down.

Right, that page doesn't even have the capability of reporting this outage. Kind of makes it a bit unreliable as a status page, you know?

:| It doesn't make it unreliable as a status page showing the status of those things it's intended to show the status of.

No, of course not. It makes it unreliable as an indicator of the Apple ecosystem's platform status. Which would be fine, were this not the only such status page Apple provides.

I'm saying it's lacking, not broken.

The page is meant to track consumer services.

Interesting, it looks like the WebObjects application(s) has actually been undeployed.

I poked around the main developer.apple.com for a bit until I got here: https://developer.apple.com/programs/start/safari/enroll.php...

Clicking the "Individual" button on the bottom throws up a WebObjects error: http://i.imgur.com/xcfZ5ip.png (Yes, that's probably the smallest I've ever made my Safari window. You're welcome, mobile readers)

Well that is a bit interesting. The message from AppleConnect (being the auth platform that is behind the"Apple ID") indicates that the "Developer" application has been revoked and is no longer allowed to auth with AppleConnect.

Perhaps there was a security breach?

I'm surprised they're allowing the application servers to respond to public requests while under maintenance.

Or it could be that the shared storage that the apps are likely hosted on could have a problem.

Right in the middle or renewing the our account. I need to activate the new license and, since I can't, people can't download our app from the store. Talking about fail...

No, waiting until the day it expires before renewing is fail :)

You can make a phone call to Apple and they will extend your account for few days if you will explain the situation.

You waited until the last day?

Well it was 2 days before... (I actually did it early but renewed by mistake another account we have)

Doh! Been there. But, I renew a month in advance. As soon as I get the reminder because I don't want to deal with issues, like this.

Argh! I'm in the same boat. I'm 2 days from expiration and I have the activation code. I just couldn't do it at work and forgot to update my home computer Xcode the one evening that I had a chance before this outage.

Looks like you are good with apple as they extended your account's renewal date due to the outage.

On another note... This is getting strange now.

I'm grasping at straws here, but maybe it was a security breach?

My first thought also.

Why? There's really no reason to think that. Of all of the things that can go wrong - power, equipment, network, etc - you pick 'security breach'... Not impossible.. but not the most likely either.

If something goes down without warning and there's no response from the organization about why it's down or when it will be back up, security incident is a natural line of thinking. Especially considering it is Apple, they'll likely keep quiet until they know exactly what is going on and how they will resolve it before saying anything on the matter.

If it was just power or network, they'd have it back online pretty quick. This seems like they're taking time to investigate something.

I'm guessing we will never hear anything regarding the matter.

If it was a security breach, and any data was leaked, they have a responsibility to inform anyone affected. At least in California http://oag.ca.gov/ecrime/databreach/reporting

If it was a breech, whether we hear something about it depends on what was accessed and what California's laws have to say about data breech notification requirements.

Power, network, etc. have redundancy. Security breach is a show stopper.

Except for the AWS EBS outage a couple of years ago which had nothing to do with a security breach.

http://aws.amazon.com/message/65648/

Cascading catastrophic failures do happen that don't involve security.

I'd think that Apple would be happy to tell us it was a simple power/network failure were that the case.

Sudden shutdown of vital service + silence = most likely a security prob.

I'm not saying that's what it is but it seems likely.

Note that the AWS writeup above didn't get published until a week after the incident. Maybe they don't know enough of what is going on yet (hence why it is an outage in the first place, because something is breaking in unknown ways) and they are focusing on fixing it vs. tweeting status updates without any useful info every 15 minutes...

Silence really doesn't imply anything except that the responsible parties are busy fixing things and that there may not be hands free/allocated to communicate with the outside world.

IMO not communicating with the outside world is a fail, but I've seen delays in this when small teams suddenly get swamped.

I'm sure that the PR department is not busy fixing this technical issue.

PR isn't the best department to post updates regarding technical issues. The technical people who understand the issue are, and they are possibly busy fixing the problem right now.

Apple is a small team?

Apple teams are much smaller than is often assumed.

OTOH they outsource a lot. From the people I've talked to at vendors "they (Apple) just don't fucking care" is what I've been hearing on this incident, which makes me think it's nothing technically interesting.

Can you expand on what they outsource? Do they outsource any of their development?

Various network/hosting services; I only know ops stuff, not how they do software development.

> I'd think that Apple would be happy to tell us it was a simple power/network failure were that the case.

Really? With the multi-region AWS failure, Amazon was very reluctant to even confirm it was happening for a while, and didn't explain it for weeks afterwards.

> Sudden shutdown of vital service + silence = most likely a security prob.

Or alternatively, "oops, how much customer data have we accidentally irreparably destroyed?!" IIRC in the aforementioned AWS incident Amazon only really started talking when they had an idea of how many EBS volumes were irreparably corrupt.

"Multi-AZ, not "multi-region" outage. (Big difference.)

When has Apple EVER told us what the cause for an outage was ?

You're just making up a conspiracy theory here.

Should have redundancy. You'd expected it to. But mistakes happen. I recall an incident where a data center found out that their two "redundant" incoming utility power feeds were really wired to the same source only when the power failed.

Plus its Apple, who have proven themselves as amateurs at anything in the cloud.

Apple has just updated the page stating that memberships due to expire today have been extended and apps will not be removed from the App Store. They have also apologized for it taking "longer than expected"

I also just received an email to this effect even though I have already renewed.

Also the iTunes connect mobile app seems to be working still.

iTunes connect works fine, just the dev portal is down

Is it still possible to provision devices through Xcode while it's down?

The message they've went with is very strange:

"This site is undergoing maintenance for an extended period today."

Normally they give a few weeks warning for maintenance.

I think so, yes. I provisioned a few devices yesterday through xcode during the outage and it updated my provisioning profiles successfully.

If it was planned maintenance, it probably would have happened before WWDC and the iOS 7 betas.

Why? They do maintenance at different points throughout the year.

Wild speculation. But it would make sense to schedule maintenance for a relatively low-traffic period.

I predict the entry of a new phrase into our lexicon

Bad antenna = "You're holding it wrong"

Outage = "Extended Maintenance"

http://i.imgur.com/14K9ogF.png

This is typical though. You don't really expect "Holy cow! Something is screwed up!!" from Microsoft, Apple, AT&T, Dell, Ford, etc, do you? Sassy/cute messages like that are the domain of Github and Twitter.

Nothing like discovering your recently released app crashes on iOS 7 and being unable to download the SDK to find out why, or fix it.

Is that really so urgent? The majority of the people using iOS 7 are developers, understand apps will be buggy under it and wouldn't leave a bad review. I'm sure an extra few hours wait won't make too much difference :)

I don't know about that. There's a huge community of non-developers who, for reasons I cannot comprehend, love to put beta OSes on their phones. I'd wager they outnumber legitimate developers using the beta by a hefty margin. And a lot of them don't understand the implications of doing this.

Stories of unjustified bad reviews being left by users like these due to poor behavior on iOS 7 are really common in the community.

> There's a huge community of non-developers who, for reasons I cannot comprehend, love to put beta OSes on their phones.

I know of some affluent high-schoolers who seem to consider it prestigious if you can show off a prerelease iOS on your phone.

Yeah I had a flatmate who was a complete Apple fanatic and would torrent beta-versions of all their software, so he always had the cutting edge revision of OSX and iOS, even though he was not very technically inclined. He just liked having the newest and latest.

If you're a non-developer and running a beta OS, you should know what you're getting into.

At the same time, I do feel sorry for devs getting bad reviews because of compatibility issues. I haven't had any myself, but it's probably only a matter of time.

I'd think that Apple would prevent someone using a beta OS from ranking apps. Their disclaimer says to run it on equipment dedicated for testing. That would preclude everyday apps, and the need to leave ratings.

Does the App Store block reviews from beta OS users? Maybe what needs to happen is if you load a beta OS on a device, it should block reviews from any Apple IDs that were associated with that device until the OS is generally available.

After all, they do warn you that you should never install a beta OS on your daily use device and they are for development purposes only.

No, they don't, and I'm among those who have received negative reviews from people running beta releases in the past. There's absolutely nothing you can do about it (flagging a review has never had any success for me).

We've got a rendering issue on iOS7, we knew about it from the numerous non-devs who demanded a fix immediately!

Nah, being able to develop applications for the largest growing and well known ecosystem since Windows 95 is not a big deal if you live under a rock and eat grubs. In fact, I can't remember the last time software had a significant impact on the greater physical world since 0.223μs ago. Too bad mobile applications on walled garden platforms never caught on, I was hoping to witness all the fun drama and obvious security and trust implications play out inside the most complicated civilized social structure in recorded human history. But that won't happen now that Apple got bored with all the profit they get from being the new masters of the information universe and decided to shut down iTunes app store thingy.

Sorry, but what's that got to do with iOS7? Parent wasn't saying that the dev centre being down isn't a bad thing, just that iOS7 incompatibility errors aren't high on the priority list yet.

The first to market with support for iOS 7 are obviously going to have a huge advantage over other App developers. the SDK being down for every developer without access to alternative platforms for their dev stack are now put back on schedule plus the length of the outage. Again, first to market in such a large market is a huge advantage, providing social and technical momentum, making or breaking careers and enterprises for those who depend on Apple's generous returns. Entire companies can be birthed or go splat due to variables like this. It's a risk they app devs are aware of, but that doesn't mean it isn't a Big Deal.

I hope it's something good. But, my guess is it won't be anything noticeable. Replacing that horrible Jive forum software would rock as it's awful.

Anyone else getting strange emails to Reset Apple ID Password? I received two today

I've gotten these twice in the last two weeks.

the maintenance page has a new update just now saying this

We'll be back soon. We apologize that maintenance is taking longer than expected. If your program membership was set to expire during this period, it has been extended and your app will remain on the App Store. If you have any other concerns about your account, please contact us.

Outage? Or Maintenance? These are two different things.

Any downtime in a cloud service is an outage. Any attempt to term it anything else labels the organization as amateur.

The page reads maintenance. But it is a service interruption, wether planned or not, it is also an outage.

Without any kind of email or notice to developers, it's an outage. A big one.

I'm told this was scheduled.

It's hard to believe that something this long would have been scheduled but that they wouldn't have told anyone about it.

I find it curious no apple rep is here, commenting on this post.

Apple typically doesn't engage with the developer community outside official channels, do they? I don't remember ever reading a post here by an Apple employee in an official or even semi-official capacity.

Really? When have you ever seen Apple reach out to the outside tech community, ever?

Every now and again an Apple person will pop up on the Xcode mailing list to tell people to submit a bug report on Radar. They sometimes also reassure all that, yes, despite all evidence to the contrary, bug reports DO get read - though I note that they are always extremely careful to promise no more than that.

They sometimes do, even if you don't count shutting down the operations for a week every WWDC. Examples:

* http://www.macworld.com/article/1167247/cook_apple_planning_...

* http://daringfireball.net/2012/02/mountain_lion

Unless by "outside" you mean those they don't reach out to, but that would be true by definition and so fairly moot.

Apple employees are not supposed to have any real online presence. Kinda sad that anyone wants to work there really.

Seriously? This is _Apple_. When the multi-region AWS outage occurred a while back, Amazon was unwilling to talk about what had happened for a week, and they're a rather less secretive company.

Finally someone has chimed in on this. I've been trying to update my provisioning profile since yesterday.

Not sure where you got 28 hours (35 mins ago) from when it hasn't been even 27 hours yet (at this moment).

I got it... Took a flight from a MDT city to an EDT city and forgot to set their watch.

Apple’s Developer Center Offline for 32 Hours; Compromised ? Details : http://thehackernews.com/2013/07/apples-developer-center-off...

It's interesting I was trying to download xcode commandline tools for setting up my new mac, the website was down but you can still download it in xcode download panel. (assuming you already have xcode installed).

It's been down at least since yesterday afternoon:

https://twitter.com/Joe_Wegner/status/357956388833599488

It's been down since Thu, 18 Jul 2013 14:26:42 +0000. https://twitter.com/iOS_DevCenter/status/357869041005760512

https://twitter.com/search?q=apple%20password%20reset&src=ty...

The WWDC vids are in an app this year, not on iTunesU. Te an phones home to validate that you are a developer. It too is inaccessible now since dev center is down is down.

Just when you need to send an adhoc build.....

Check out Nomad, it's by Mattt (of AFNetworking fame). http://nomad-cli.com/

This uses web scraping of the developer portal behind the scenes, and as a result the Cupertino module is unusable during this outage.

Or download the command line tools! The downside of all this nice Xcode integration.

http://techytoolbox.com/clt/xcode452cltools10_86938211a.dmg

The URI is from https://github.com/jwaldrip/dev-setup/blob/master/setup_envi... , which looks trustworthy enough. Use at your own risk though.

Thanks! Looking for a later version, but appreciate the help.

If you have another machine with the tools installed, you may find the DMG you want in ~/Library/Caches/com.apple.dt.Xcode/Downloads

This is a major bummer. I wanted to push out some more TestFlight builds yesterday and have been in limbo ever since.

this is screwed up too http://devimages.apple.com/

It has always been.

anonymous has been hitting targets listed in Prism presentation of late - could be one of the anonymous' casualties?

Even if that were true one would think that with all the resources Apple has they would be able to handle a DDOS. It would be nice if this was intentional but sadly judging by how long its been thats very unlikely so DDOS is possible.

Forums are now down as well.

Forums were down since yesterday for at least 24 hours as well.

I was able to access them for quite a while even long enough to read complaints about the outage. Wonder if they just turned them off to prevent all the would be posts about the downtime

Anyone have an Xcode 5 dmg?

I do, but how can I be so sure you're a AAPL developer :).

I work at Sincerely :0) I'm the only one with iOS7 on my phone but need to build to it right now and don't have xcode 5 :/ I suppose I could wait

I got XCode 5 but none of the documentation installed. :(

its back for me

[edit] nvm, that was the marketing developer site.

It's working now: https://developer.apple.com/

lol no https://developer.apple.com/membercenter

Deleted comment

That comment wasn't constructive the first 2 times.

Wait, does this mean the entire Cloud is down? I can still access Google and Amazon.

Correct. The entire cloud, aka internet, is down. The only reason you can access Google and Amazon is because of the latent electricity in their data centers' cabling. It's just a matter of time before they go dark too. Good luck.

Reply on news.ycombinator.com