Apple Developer Center is Fully Restored | Hacker News Reader

Apple Developer Center is Fully Restored

developer.apple.com

46 pointsnimeshneema46 comments

I wonder what Apple's server infrastructure looks like? They aren't programming it all in WebObjects [1] are they? Do they use linux and/or freeBSD? Of all the things Apple does, their "cloud" seems the weakest product.

Apple is the only big software company that doesn't develop server-side technologies. If I was the boss of Apple I'd standardize on FreeBSD and Golang today! But, I'm not... obviously :)

[1] http://en.wikipedia.org/wiki/WebObjects

I've brought this up before, but it's a mix of everything, and all pretty bad.

> If you've ever poked around with the way that Apple's website works, you can see that the entire place is a huge mess. There's old servers running ancient (pre-2004) perl scripts alongside the brand new iCloud gear. I can't imagine how the authentication for AppleID is working as login details still work on the ancient pages (think pinstripes and glassy buttons). Depending what URL you hit, the webserver is using php3, php4, perl, python or maybe WebObjects (java).

> At one point I wrote a scraper that was targeting one of their product pages, and kept getting random, unexplainable results. It turned out that one of their product areas was behind a round-robin load balancer, with three completely different apache versions on each server. The page was dying on one but not the other two. In the end I just had to repetitively scrape until I hit a good response.

https://news.ycombinator.com/item?id=6078854

Wow, that's exactly what I was afraid of... You can tell it's absolutely not their priority. It's more of an afterthought. "Oh, yeah, btw, we need some servers as well, but first do all the other stuff."

> Apple is the only big software company that doesn't develop server-side technologies.

iCloud storage, iTunes Match, iMessage, to name just a few. If you stop to think how these all just work across devices and desktops, and at what volumes, you'll realize they've got some server chops.

I meant they don't create any server-side language or framework... They will have lots of infrastructure and code, no doubt.

If you've ever used iTunes Match, you'd know it's a stretch to say that it "just works".

You can say the same thing about iCloud. I'm not a Microsoft fan-boy, but I have to say that SkyDrive does what I expect it to.

Sometimes. I've had a couple of files deleted which I was not happy about. For that alone, they can get screwed.

They mostly run on Microsoft Azure and Amazon EC2.

They also don't really work. iMessage is unreliable at best, and iTunes match just makes home taping look preferable.

If that were true they wouldn't need the large data centres they're building.

My understanding, from when the references to Azure and Amazon CloudFront (not EC2) were first discovered in iCloud traffic, was that Apple was using Microsoft and Amazon's CDN services, amongst others, to serve static content.

I've seen no indication that the core guts of iMessage et al live anywhere but inside Apple's own data centres.

I based those statements on my own observations, though I've never had the time to go and dig any deeper. iMessages push layer did seen to sit on EC2 though, which would imply that other parts of it did too.

So the whole place went down a few days before we were ready to launch. We need to upload an iPhone app to iTunes Connect. We have been prevented from doing this because we needed to accept some new T&Cs on the member centre. Despite iTunes Connect being up, we've been stuck. Apple support were unhelpful - we've just had to wait it out. Now finally I've been able to get on the member centre and accept the T&Cs, but there's clearly some delay in iTunes Connect realising this. I still can't upload my app.

Many more profanities are being hurled at Apple this morning.

Sorry but when you base your businesses on walled gardens you are forever at the mercy of others.

There is nothing to complain about here; you've just got to suck it up.

The business is not based around mobile, but if you're going to have any kind of mobile apps - however incidental to your core product - you have to put up with it.

It took over 3 weeks to restore everything. I hope Apple does some sort of post-mortem (at least something basic). When a security breach takes you down for this long it was either a very serious breach or your security and infrastructure was shit and your customers deserve to know how you've remedied that.

..if microsoft's or google's services were down for this long hell would freeze...

just saying that apple seems to be getting treated better for no reason

You mean, if Microsoft's or Google's developer services were down for this long. I honestly don't think people would raise any more stink than they did with Apple.

On the flip side, Apple took a security issue very, very seriously, and for that they should be commended. I mean, obviously better if it were found beforehand, but clearly someone outside of the people in charge of these web properties had a WTF moment and life became unpleasant for some people for a while.

Most firms would have duct-taped the specific section exploited.

> On the flip side, Apple took a security issue very, very seriously, and for that they should be commended.

Though they were warned before the breach occurred, right? They didn't take it seriously until well after they knew they needed to fix it.

Great Saturday morning news, I've finally able to run my new iOS app on my iPhone! I had to accept new T&C before getting certificates/provisioning-profiles, and this has been impossible for weeks.

The profiles center has been a online for quite a while now.

> I had to accept new T&C before getting certificates/provisioning-profiles, and this has been impossible for weeks.

If you can't accept the new T&C you can't get into the profile center.

But the agreements portion wasn't. My company's enterprise license was supposed to run out a week ago. Luckily they extended it, but the hub of everything - the agreements server - was down.

ugh, what a mess.

Argh, it's silly that it takes some time for a membership renewal to take effect.

Why isn't this valid criticism? Buy a digital membership anywhere else and the effect is immediate. Buy something on iTunes and you get the product immediately.

Let's pour a cider on the ground for all the fallen evenings and weekends that were surely sacrificed by our industry comrades to accomplish this restoration.

For fucks sake. If they'd done a half decent job to start with, they wouldn't be there.

They fucked up, now they're paying for it.

Surely you don't believe that everyone who busted their ass to fix the problem was also responsible for causing it?

If Apple had a security breach, and it seems like a massive one given the circumstances, are they not required by California law to report it to their customers?

They did report that there was a breach, that some personal data about developers (but no credit card info or passwords) was stolen and that they are restoring the developer center.

That's more than enough reporting. If you want a full post-mortem, that's likely not something you'd get from a public company as big as Apple

AWS always gives full and detailed postmortems. I wouldn't say it is a trait of big companies specially not to disclose to customers, but it is something I would expect Apple to probably do.

You wouldn't get a full post-mortem from a company like Apple, because they are wilfully intransparent. My employer (Spotify) for example posts somewhat detailed post-mortems[1] after big outages - doing this always comes with a spike in job applications so it's positive in several ways.

Another comment mentioned that AWS post-mortems are also detailed and public, they don't really have another choice because their customers have their infrastructure running on AWS - so they want to know what happened and not be left in the dark.

[1] http://labs.spotify.com/2013/06/04/incident-management-at-sp...

It is not for me. When I try to Join SAfari developer Program it is redirected to this URL:

https://developer.apple.com/iphone/urlRedirect.action?mode=e...

From the action URL, this seems to be the page you want — https://developer.apple.com/ios/enroll/individualEnrollmentI...

Hmmmm, after I accepted the new terms & conditions, the iTunesconnect still ask me to accept it in Member Center. I didn't expect to have some sort of delay between the changes in Member Center and iTunesconnect. Hope it takes effect soon. Sigh.

I was wondering why I was unable to access any of the services that were supposedly back online. Now that the Member Center is back, it turns out there were new Terms & Conditions I needed to accept first... argh.

My account is still subject to expiring even though I did a renewal before things went down - should have eleven months at least.

Anyone else got this issue?

Not not often one to praise Apple, but I'm really impressed that they spent the time they did to fix this. Apple are known for their strong brand and taking down the dev centre for so long must have been an awkward decision between the marketing side and the technical one.

I don't know the extent of the problem but taking down an online service for weeks is very uncommon. I guess (speculating a bit) that Apple could have applied a quick fix to the problem within a day to save their (short term, at least) brand appearance. I think "just getting it secure enough" is the most many people would do if an important service was down. It appears that Apple took the time required to deploy a proper fix, prioritising security over shot term wins.

It would be interesting to see what someone like Microsoft or Amazon would have done in this situation, or what Apple would have done if it was all of iTunes instead of the dev centre.

Does anyone know of a centralised database of security breaches? It would be great if there were a standard (like CVEs), which included how well the breach was dealt with (eg informing public, mitigation, prevention, changes made). If there isn't one and there's interest here then maybe I'll start one.

You're impressed they spent time keeping their cash cow alive? No offense to the developers, but there has been very little to no communication. I'm magically waiting for the guy with the best F5 time on the dashboard to get my news? Again, no offense to the developers, but there's no indication that this was not a "quick fix" by industry standards (of security) and not just "secure enough."

As usual, a dashboard of green lights does not identify uptime, as can be established by the issues people are still having with the platform.

>No offense to the developers, but there has been very little to no communication.

There's been weekly or slightly shorter than weekly updates and a 24-hour status page. Also, most key portal functionality was available not that long after the breach.

It's a little unfair to say "very little to no communication".

Communication that says they are down and that's it. I sent mail to support and they sent me a mail asking me to wait. I had to wait for a month now to accept the new license.

In my books, that is no communication. They haven't told us what the problem was at the end of the day.

You are very generous with your praise.

They did nothing good here. They fucked up and took forever to fix it. Please stop apologising for them.

Oh I'm not apologising for them, they still messed up (and I still actively discourage people from buying Apple products). I don't use any Apple products / services, so their downtime didn't bother me. Their lack of communication is bad but I'd prefer poor communication over poor security.

We know they didn't push a quick hacky fix live. I think it's quite likely (due to the resources they have) that they deployed a proper fix, not sacrificing security for a faster response. I could be wrong.

Reply on news.ycombinator.com