Our service is down because Msft Azure is down. This is how we chose to react. | Hacker News Reader

Our service is down because Msft Azure is down. This is how we chose to react.

modern-products.tumblr.com

35 pointsroee60 comments

This is a nice gesture, but I prefer the services I use to take responsibility for their provider and technology choices. Your customers don't care that Azure is down, they only care that they can't do whatever they were trying to accomplish. This isn't Azure's fault. It's yours. Don't pass the buck.

And if all they said was they were experiencing difficulties you'd see people complaining that they aren't giving any indication as to what the problem is.

The site is down because their provider is down. This is, as someone else stated, a fact. They told you that and that is really all they can do. They have to wait just the same as you do, for the same thing.

> This isn't Azure's fault.

Yes it is. The alternative is to pay more (a lot more) for projects to host all their own servers and storage and backups. That would be the end of a great many small to medium companies.

It's a myth that dedicated hosting in several geographically distant data centers really costs that much more than cloud computing. With a little bit of care, it's quite easy to get very good value from such hosting, with far more control over it. Outages like this could be easily avoided, or at the very least dealt with much quicker.

It would not be the "end of a great many small to medium companies" were they to use dedicated hosting. Hell, this is how it was commonly done up until the past 2 to 3 years, and small and medium companies thrived quite well.

In fact, after the cloud outages of the past week, and a few other recent outages, more traditional dedicated hosting keeps on looking better and better.

We actually pay for georeplication on Azure. Here's the official response from them:

"Why didn’t we just fail over? We do have geo-replication for Windows Azure Blobs and Tables, where the data from US South is geo-replicated to keep another replica set of the data in US North. We have chosen at this time not to failover, since we believe we can bring back the primary storage stamp in US South in place. One of the advantages of recovering in place is it avoids losing the Windows Azure Queue data in that stamp, since Windows Azure Queues is not being geo-replicated at this time (we are working towards turning that on)."

How is this different from, say, a coffee shop saying "sorry, we're closed because the electricity went out"?

I guess the difference would be that there is usually only one electric company that can supply you with power.

A coffee shop has very little equipment, and what they have tends to be isolated from other utilities (such as gas): they could easily run their shop off a generator (mate even a battery for a short time).

They're not passing the buck, they're voicing their unhappiness. It's important that this happens, because it damages the reputation of cloud providers that go down a lot (like AWS' US-EAST). Being down for 24 hours is pretty impressive, in a bad way.

There's a time and a place for voicing such displeasure. When your website and service is unavailable, it is not the time, and your maintenance page is not the place.

I don't think this is a server-to-customers message, but rather a programmer-to-other-programmers message.

I tried Microsoft Azure for a month because I received free credits via their BizSpark program. I can confidently say their uptime is the worst I have ever seen. Nearly daily servers would loose access to disks and switch to read-only mode. Also, they lack any ability to snapshot backup servers, which is a basic requirement of a cloud platform.

I know its early, and they are just starting, but I just don't have confidence in Azure. Stick with AWS, Linode, Softlayer.

When was it? We actually have great experience and great uptime with Azure. It's a very unique case for us.

Are you running Linux or Windows? This was June/July.

We're running on their table storage from web roles and worker roles (their platform-as-a-service)

I have a VM on azure, and no downtime in last 3 months. Also, didn't notice any problems with disks.

I'd second this. I've found that VM's (especially Windows) run even more reliably than what I had on EC2 (sometimes the machines just got soooo slow on EC2 for no apparent reason)

When a server gets slow, log in and look at top or vmstat. There's a field called st that stands for steal time. If you're using a micro, you have higher peak CPU available, but lower baseline. We found they were notoriously bad about the average CPU we got and our default is now a small (we also moved to non-ebs backed instances for the primary drive for other reasons, which is unavailable on micros). If your steal time is high, you're over your CPU budget and it's time to look at a bigger instance.

This made me think of the Netflix outage the other day. When people like my non-technical parents say things like, "did you hear Netflix went down," they don't care that it was Amazon that really went down. I don't use Soluto, so I don't know their customers, but I doubt they care that Microsoft has dropped the ball.

It's a chance we take when we deploy to the cloud, and services like Heroku only compound it because that's an additional point of failure. I'm not sure a giant "it's their fault" finger point is the best way to handle a problem, but I assume they feel pretty powerless, so finger pointing is an option.

Good luck resuming operations. Hopefully you won't loose too many customers.

A classic case of: "you can't outsource responsibility".

When can we move beyond blaming a cloud service and just owning up to the decision? Soluto chose to use Azure and knew they were taking a risk. It's their fault for choosing to use Azure, and unless they signed for a 100.000000000% uptime guarantee (which I'm sure they didn't, given no one would give such a guarantee) they have to own up to any faults.

This extends to those services that blames AWS as well. It's not a good habit, and customers (like we saw with netflix) don't care -- all they know is that their service is down.

Azure offers a SLA with a 99,95% uptime guarantee for computing instances and a 99,9% uptime SLA for storage. If they do not meet this SLA you will get your money back.

You can read more about it at www.windowsazure.com/en-us/support/legal/sla/

They've already blown that SLA - for the year. Now how much of my money do I get back?

All of it I think?

SLAs are worthless, because just getting your money back does nothing to repair the damage to your business. It's like if someone in your family died from food poisioning and the restaurant gave you back your $10 for lunch.

No one's "blaming" anyone. The expectancy is that cloud providers work their asses off to fix stuff that gets broken, which is the case.

"Our service is down because Azure is down" definitely sounds like scapegoating (regardless of whether it is the case, Soluto should have a contingency plan)

That's not scapegoating, it's pointing at facts. Sad as they are. When you have a complex web service, you rely on a provider. That provider can go down sometimes, and it takes you with it. Again I'll point to Netflix and AWS, they waited until AWS were back up, they didn't "restore a backup" somewhere else.

It wouldn't be scapegoating if their site was still up and their service still functioning, hosted by some alternate provider, for example. But since their site and service aren't up, I think it can be seen as scapegoating.

What is the extent of this outage? There isn't much info on Twitter or elsewhere. The Azure service status dashboard shows an issue impacting "Storage [South Central US]". Is that service like S3 or EBS or something entirely different?

Azure Storage is for blob storage (like S3), non-relational table storage (like SimpleDB), and queues (like SQS).

I didn't know what Soluto was, so I went on your main page to find out what it was while your service was down. All I saw was "Service Disruption :( We are experiencing problems with our cloud infrastructure More information..." Nothing else to click on but that one link for support. So when I clicked on it, still didn't know what it was and so I scrolled all the way down and clicked "About" then yet again I was greeted by the same alert. I understand your service is down, but does it have to take your whole site with information on your product with it just to show your service is down?

Anyways my 2 cents.

Regards.

That's right. It's double-bad when things like that happen during the weekend so time-to-reaction is slower. All will be better tomorrow.

Are you trying to tell us that proper automated failover systems take the weekends off?

I'm trying to say we were not visually prepared for this, and that getting designers and coders to work on a weekend is not something fun. I see your sarcasm and get it, but really it's out of place.

Maybe a noobish question, but in the future, is there anything other than need for preparation, that prevents you from having a backup spun up on say AWS or another service provider?

It seems like a prudent step to take.

It's the entire service, not a bunch of files. When Netflix went down due to AWS outage, could you image them just "restoring a backup" on rackspace and running just like that?

The core service, sure. The problem is that Netflix accounts for something like a 3rd of US bandwidth consumption. Not many services can sustain that. Netflix also has to manipulate a giant library of assets. Doesn't sound like this is part of their service. A normal company should be able to fail-over.

I have been experimenting a lot with Google App Engine. Failing over from a PaaS like that is not easy because your app typically relies on a bunch of proprietary APIs. However, it is possible, provided you keep right replication and architect the application accordingly.

You also don't need to fail over 100% of your app. I don't have a very clear understanding of what Soluto actually does (sounds like a GotToAssist competitor), ability to fail over the part of the system that facilitates the remote desktop experience would be more important than administration or creation of new accounts. (This is a crude example.)

I've got several services (both consumer and commerical) running on Azure and right now I just see some issues with storage in the south. Web/Worker, VMs, Sql, etc. are running just fine it appears. Any pointers on where I should be looking for potential issues?

I can recommend trying AppHarbor - we're striving to deliver a better .NET and Windows cloud platform. Feel free to shoot me an email (rs@appharbor.com) if there's anything I can help with.

AppHarbor is fantastic. I can't stress this enough. I've deployed three MVC3 applications on there already and they work effortlessly.

My workflow is:

1. Write code on dev machine.

2. hg push to BitBucket.

3. AppHarbor hook reads new push from BitBucket and compiled on their service.

Simple quick and easy deployment!

Once thing I wish they would offer is free domains like Heroku does. Hopefully this is a feature they enable in the future.

I've heard a lot of good things about AppHarbor. I still avoid .NET and Windows, but if I had to host .NET or Windows stuff somewhere, I'd go with App Harbor.

Its just too expensive, you cant host your custom domain name on free instance. And there's nothing between 0 and 50$. I'm waiting for prices of azure web sites, hopefully it would compare to shared hosting prices.

You can add custom hostnames to the free instance for $10/month

Can you explain to us why such basic functionality is so expensive?

Is there a legitimate technical reason for the cost, for example? Or is it subsidizing some other service? Or is it that high just because some people are foolish enough to pay that much for it?

The paid plans already include hosting on a custom domain. I find it surprising that they let you pay a la carte for custom domain support on the free plan as well.

good luck changing your email address ;)

I rely heavily on my spam filter :-)

I love this, it's refreshing. Every time something goes down, I see a flood of hate and a torrent of comments suggesting everyone move somewhere else. That, coupled with HN's hate of MSFT, makes this one of the most refreshing posts I've seen in some time. And sensible.

Respect to you, my friend :).

For those who are curious

Cached copy of the front page: http://webcache.googleusercontent.com/search?q=cache:0vf1uUF...

Their support site: https://support.soluto.com/home (not down)

Wikipedia page: http://en.wikipedia.org/wiki/Soluto

Soluto alternatives: http://alternativeto.net/software/soluto/

Wikipedia says that one of their remote features is: "Defrag the hard drives"

Someone pushed the wrong button there? ;)

Seriously, that just reminded me of running Windows back in the old days. You had to make sure to have the latest AV, do system upgrades all the time and of course defrag the drives... yay

One of the FAQ answers on the page linked to by the "More information..." link on their outage page states, "Cloud services like ourselves someone (sic) experience such outages, it's part of the modern web."

I can't believe that it actually says that. It seems extremely unprofessional to me to make a statement like that.

Downtime like this is not acceptable, regardless how the service is hosted, and who is providing the hosting.

We didn't put up with this kind of outage with the "old fashioned" Web, and we shouldn't put up with it from the so-called "modern" Web, either.

Perhaps this is part of your current problem with Azure, but is it possible to retain some part of your frontpage during an outage like this? Or failover to a static page with a basic description of the service? Your support site seems to look ok.

I only make the suggestion because I was not aware of Soluto. I went to your homepage and see the outage notice and a link to support. I still have no idea what your service is about.

Perhaps not the best time to be spruiking your site and service, but any-publicity-is-good-publicity etc.

my point exactly.

Good point :)

Looks like Azure is having problems in the US south-central region [1], but fine everywhere else. The fact that they don't mention their region suggests they may not understand the implications.

If the status page is accurate, their downtime is just poor planning on their part. Same goes for Amazon. If you're stranded in one region, you're asking for downtime.

[1] http://www.windowsazure.com/en-us/support/service-dashboard/

Maybe the world is ready for RACSP (Redundant Array of Cloud Service Providers)

That's too close to RackSpace. Choose another name :)

My impression of that page was not very good TBH. The words are kind, but for me at least, the message boiled down to:

"We're down." "It isn't our fault." "We'll be back up when someone else gets it fixed."

You must be at least _thinking_ about how to avoid/limit this impact in the future. Why not give the user more information along these lines, and thereby position yourself as an active participant in the process at the same time? Polite helplessness is probably not what you want to project as a business, but that is what I took away from it.

The only reported outage I'm seeing is Storage / South Central US - https://www.windowsazure.com/en-us/support/service-dashboard...

I expected to see them stop using Azure or something, but this is a good idea that's entirely opposite from what I was expecting!

Reply on news.ycombinator.com