Heroku Postmortem of June 29th Incident | Hacker News Reader

Heroku Postmortem of June 29th Incident

status.heroku.com

28 pointstimf20 comments

The "What we are doing" section seems pretty weak. The only substantive thing they say is "we have produced new tools which enable us to more expediently relocate database services from a failed availability zone."

How exactly are they planning to deal with the larger Cedar difficulties? Are they going to eliminate their dependence on ELBs? Go multi-region? Developers need to know this to decide whether to continue with Heroku or build their own platform.

Compare/contrast to https://status.heroku.com/incidents/151 where they did talk about moving to multiregion, etc., but then never executed on it, which is probably worse. Maybe they will underpromise and overdeliver.

Personally, I'd shitcan EBS to the extent possible, RDS I never would have used, and it looks like ELB is non-viable as well.

If you're a large site (particularly a PaaS) on AWS and care about availability, you need to have spare capacity in your region (using RIs, like Netflix does) to cover when a single AZ disappears, and your own external to AWS load balancing (not dns based), with your own per-AZ subsidiary load balancers (nginx or whatever) running within EC2. You need a robust database layer, ideally multi-region or AWS+nonAWS, but that's more site specific.

Going multiregion is the next step, and the above is an essential part of getting to that point.

What's an RI, please? That one's hard to search for.

Reserved Instance (use the low duty cycle ones for availability)

One subtle but important reason to use cross-region failover is that the network latency between the regions can prevent many casual or accidental dependencies between instances (if you configure instances in two regions to use the same database server, latency can cause the distant region to perform poorly).

This is why it's really hard to get cross region failover to work. Because you really need to make them independent.

You're describing a premature optimization. Performing poorly is better than not performing at all.

> Approximately 30% of our EC2 instances, which were responsible for running applications, databases and supporting infrastructure (including some components specific to the Bamboo stack), went offline

Combined with the incident report from amazon, does this mean that 30% of Heroku instances were in a single availability zone? That would be troubling.

Why is that troubling? There are currently 5 AZs in the region (there were fewer when they designed their service), so the best they could possibly do is 20% of their instances in each. Losing 20% of their instances instead of 30% isn't much of an improvement.

What they need is failover between regions/clouds and external monitoring, not a slightly better allocation of instances between AWS AZs in one region.

I don't use Heroku, but I would have assumed that they exist in more than one AWS region.

They don't.

According to this tweet, they are in 3 AZs. https://twitter.com/heroku/status/219937314749677568

3 AZs all in the US-East region. That's consistent with what we're talking about.

One of their suggestions is to have a follower of your DB and fall back to it. When they put the API in read-only mode, would I have been able to promote any followers?

My questions are:

- Why aren't they committing to using geographically dispersed AWS instances?

- Why aren't they leveraging Salesforce's infrastructure at all?

You may not have noticed the 2x multi-hour outages Salesforce has had in the past 2 weeks, at least one due to "power problems", the more recent of which took down their fucking status site too.

My point wasn't that Salesforce's infrastructure is reliable (although they arguably have more pressure than most companies to be so), just that there is a separate resource they could be utilizing as a backup/in addition to AWS that they currently are not using.

Presumably if they were having to deal with a power outage you could reliably assume that that fucking status was set to "not fucking"

Committing to geographically dispersed anything is a hard problem. How will apps perform when their front-end is in Virginia and their back-end is in California? How will fail-over work when it involves moving a 100GB database from Dallas to Washington, instead of between racks in the same building over a private gigabit network?

I agree that it's a difficult problem, but it's Heroku's obligation to solve those problems. You can't position yourself like this:

> Heroku takes full responsibility for your app's health, keeping it up and running through thick and thin

and not commit to solving hard problems like geographically dispersed EC2 instances. Resource constraints, to be fair, aren't an acceptable excuse at this point. They aren't a floundering startup. They're a part of a multibillion dollar company.

Relying on AWS' API to mitigate an AWS failure seems dangerous. A fundamental catch-22 with on-demand provisioning/configuration.

Reply on news.ycombinator.com