Hacker News Reader

Comment by colinbartlett | Hacker News Reader

This is manifesting itself as downtime for a lot of companies, including Heroku: https://status.heroku.com

If you want alerts on this sort of thing, my side project StatusGator https://statusgator.io will alert you when services post downtime on their status pages. My dashboard blew up this morning with a ton of red and yellow as soon as Amazon started flaking.

Edit: I suppose it's time to invest in a multi-region setup. Since StatusGator is hosted on Heroku in the US-East region, it is in theory affected by this problem though so far is still up.

View on HN

From Heroku Status Page:

> Our service provider is still working towards resolution of this issue. We will update when we have news, or in 1 hour.

I wonder why they don't tell that AWS is their service provider. Is it wrong to make the information less obscure?

Yes, as it's shifting the blame away from their choice - which was to use AWS.

I expect the reason is because Heroku could switch to a new provider in the future and it would be a pain to always update every reference to their provider.

Parent seems to imply there's something wrong with choosing AWS. There is not. (forgive me if I mistook the tone)

You did; there is nothing wrong - but it's a choice. I.e. your site being down is your fault ultimately, not AWS's - you choose AWS (instead of many other choices OR making something with multiple, etc).

Blaming it upstream is hiding passing the buck on your decision.

Indeed. If your site is down, it's your fault - no-one elses. Yes it may be down because someone downstream is down, but ultimately it's your choice where and how you host.

Because it is kind of awkward PR for Salesforce: come build on our platform! Oh by the way we haven't even moved Heroku, a company that we bought 4 years ago, over to it yet.

Why move it if it works fine as-is? I think Waze was bought by Google in 2013 and AFAIK, they're also still in AWS.

The PR is still awkward: "come use Google Cloud Services!" while it's not good/easy enough for your acquisitions to migrate to it from AWS.

Even Amazon.com isn't fully reliant on AWS. They're not using Route 53 for their DNS (or even for their backup DNS).

They will never be able to move Heroku of AWS. Their customers depend on being on the same cloud as other service providers

The location of instances could be an option. I imagine if their locally-hosted instances were a bit cheaper they would be the more popular option.

> I wonder why they don't tell that AWS is their service provider.

It's because Heroku's choice of vendors shouldn't matter to their customers. They see it as an implementation detail, and their responsibility to manage.

So I don't think that's an obfuscation. The people I know at Heroku all have an attitude of, "The buck stops here."

That's just stupid. Heroku rarely gives a proper technical explanation of their outages and under reports the length and severity of them drastically.

I assume this is to maintain their SLA. We really need independent third parties to record uptime for SLAs instead of trusting hosts to do it themselves.

This outage may be the last straw with Heroku for me. They've also previously stated years ago that they would end their dependance on AWS East and yet today shows that obviously hasn't happened.

Not that i think there's anything wrong with heroku's messaging here, but "we're down because our service provider is having issues" isn't really a "the buck stops here" sort of update.

Interesting that the outage in us-east-1 is causing huge issues for Heroku in eu-west-1 too - obviously a region for Heroku isn't self sufficient.

Where do you see that it's causing issues for Heroku in eu-west-1? I have a lot of dynos in that region and I don't see any issues

Usually Heroku mentions a specific region in any status updates that don't affect all regions. For example, a status update 11 days ago stated "We're currently seeing increased routing latency in the US region. Investigation is underway."

AFAIK this outage only affected dyno restarting (which may have been triggered by a number of reasons) and creation of new dynos. Perhaps your EU dynos were lucky enough to not have done either of these things during the outage?

"Until this incident is resolved, you might be unable to open new support tickets with us."

Maybe they should host the ticket system on a separate provider?

Reply on news.ycombinator.com