Hacker News Reader

Comment by jread | Hacker News Reader

It was a multi-region, multi-hour outage on Nov 18. To my knowledge AWS has not experienced a multi-region outage. That said, they are learning from these events and improving stability.

View on HN

I read that they didn't renew some certificate which needs to be renewed periodically. I also remember something like that happening with hotmail many years ago, where they forgot to renew their domain registration, and all of hotmail went off the net.

I had a friend who worked as a contractor for one of Microsoft's data centers a few years ago, and the stories he told me about how they run their datacenters was frightening. It sounded like management was out of touch and didn't really have a grasp on what was needed to keep everything running well.

Those stories leave me questioning Microsoft's basic competence in the cloud. While I'm glad to hear that they're improving their service, I just don't see them as being in the same class as AWS, Google, or even some of their smaller competitors such as DO. I would be wary of depending on Azure for anything critical.

They actually did a walkthrough of one of their DC's on video a while ago...it looked fairly impressive and well put together to me, but I've never been in anything but a small Co-Lo, so I'm not really sure what to look for.

I work on infrastructure at MSFT. Collectively we stably runs hundreds upon hundreds of thousands of machines in clusters that are tens of thousands apiece, at scale workloads that only 2 or 3 other companies total have ever seen. So, when I say that it is very hard for me to reconcile your assertion that we lack "basic competence" in cloud management with the facts of the situation, I hope you will understand why. It makes me think that you have not thought carefully about what you are saying here.

If you'd like to share the specific stories this contractor told you, I'd be happy to talk you through it to figure out where your information went wrong.

I'd like to hear these stories as well.

Not sure what you mean by where my info went wrong, but my friend worked at a MS datacenter a few years ago, so I don't know what has changed. At the time, almost all of the techs working at the DC were contractors, not MSFT employees. They were severely understaffed and getting cut back constantly. There were a few different companies doing the contracting as well, and all of them were under the threat of losing out to a lower bid company. The competent techs were getting outnumbered by incompetent techs. That and the fact that MS couldn't even renew basic registrations and certificates didn't surprise me with my poor experience with Azure.

Again, this was a couple of years ago. Also, I attended an event that MS held to show off Azure, and a large percentage of participants couldn't spin up an instance within 10 minutes, some of them didn't even start up. The MS people running the event had a direct line to some techs in the datacenter, but they still couldn't solve the problems. I talked with some MS partners/users who told me that Azure was terrible in their experience and even some MS execs admitted as much in private conversation. It sounds like Azure has improved since then, but I really have to wonder how much improvement has occured.

I take your points to be the following. Correct me if I'm wrong. (1) Morale is low among our DC technicians, (2) many of the tech vendors are incompetent, (3) one time hotmail forgot to renew a cert, (4) you had a bad experience one time with an Azure demo.

Honestly, aceperry, I'm not seeing where any of this reflects poorly on our ability to operate x million machines, and the fact that you present it as evidence that we are not "competent" at provision our cluster effectively for our customers is worrying. I am starting to wonder if you know anything at all about the trade, because literally nothing here is relevant to your case. Even the hearsay you have about the DC vendors being incompetent is not really useful here, because it does not impact our operational capacity at all.

What am I missing in your statement that should cause me to worry?

You are right - the big outage happened in November. Feels like yesterday (it was a stressful day for me). AWS outages have been more limited in scope/geo, still took some sophisticated users of AWS offline (Netflix, Instagram come to mind).

Reply on news.ycombinator.com