Tuesday's Heroku outage post-mortem
blog.heroku.com
blog.heroku.com
... and hadn't already done it.
This isn't meant as a slight against Heroku at all. They've got an incredible team of engineers. But imagine if Ricardo had said "hey, I could write a patch today that would speed up our messaging system 5x, should I do it?" the rest of the team would've said "OF COURSE!"
It reminds me of what happens to your brain when you launch a site. Even before you get feedback, somehow the knowledge that people can use it drastically changes your motivation system. Things that before seemed important are obviously not. Other things that were invisible before become the singular focus of your resources.
Maybe we should do fire drills...
* Your requests are suddently taking 100x as long to complete. Go!
* Your "runway" disappears due to an accounting error and you have 7 days to turn a profit. Go!
* 50% of people visiting the site have no idea how to use it. Go!
How could we achieve the focus and clarity that a crisis brings on, without having the crisis?
Regarding the 5x increase:
> One of our operations engineers, Ricardo Chimal, Jr., has been working for some time on improving the way we target messages between components of our platform. We completed internal testing of these changes yesterday and they were deployed to our production cloud last night at 19:00 PDT (02:00 UTC).
I could spend all my time working on optimizations to the system, or I could work on things that will bring us more money. The company would rather the latter, I would rather the former.
But here's the key: Until there's a problem, you don't always notice the small things you could do to give a huge performance increase.
I also don't think mental drills are the answer. When I find a bottleneck, I don't just sit there and think. I'm out reviewing logs and watching the actual performance of the system. Without any details as to what's happening, how could I possibly find the real bottleneck?
Storytime: Once upon a time, we had a customer who abused our system. He submitted more data than all the rest of our customers combined. I loved him, because he showed us all kinds of bottlenecks. Management didn't love him because his stress on the system would frequently cause issues for other clients. Even better, the stress he put on the system really was just like regular clients, and not some abnormal stress. By the time they let him go, I had found so many bottlenecks that we didn't have another system issue for -years- after he was gone.
Heroku sells a technical product to a technical audience. They're foundational to their clients' products. So when something goes down, there's only one option: explain, in excruciating detail, exactly what happened, why it happened, and how it's going to be fixed in the future.
Why? Because their clients can smell bullshit better than a purebred bloodhound. Too much bullshit means it's time to move on.
Beyond being the right thing to do, being accountable is essential to trust. When you fuck up, it will piss people off. That's just life – everyone makes mistakes. So you need to be the guy where people can say "Okay, there was a fuck up, it was bad, but look at how hard these guys worked to fix it. Check out their plans to prevent it in the future."
Luckily, the incentives are aligned here to make this mostly non-negotiable. When you get medical malpractice, a financial meltdown or an oil spill going on, the cover-your-ass impulses are much more compelling.
Even in those cases though, I insist we need to encourage a culture where accountability and transparency are rewarded. Because, for me, accountable guys are the kind of people I want to do business with.
I dunno much about scaling a Rails server, but for now, at least, I know the Heroku guys are the sort of people I'd trust.
Okay. I feel a bit sorry for bashing heroku here, but I'll bite.
If I was a heroku-customer then I'd feel, ahem, a bit washed by their idea of "excruciating detail".
So their "internal messaging system" triggered a bug in their "distributed routing mesh". And they applied a "hot patch".
Great. As far as I am concerned they could as well have written their flux-compensator overheated because the pixie-dust exhaust got clogged with rogue bogomips.
I applaud their willingness to talk to their customers at all. But please... either explain what was going on in a meaningful way - or just leave it at "we screwed up and promise to do our best to prevent it from happening again".
Some of us like a technical breakdown and feel warm fuzzy reassurance. If a few people got confused after the first paragraph, it's less harmful than appearing to bullshit technical users.
In less snarky words: Even facebook told us quite clearly _how_ they screwed up the other day (the config management issue). In contrast this heroku article was disappointing.
NASA reports have excruciating detail. This felt a big vague.
The problem is that you can build an elaborate maze of QA checks and still miss problems like these (and kill the company's ability to innovate in the process).
The reasons why you have (or don't have) various QA processes are much more interesting than the processes themselves.
Why? Cause everything depends on the type of problems that people have to face.
So to say that "Google gets it wrong" as the whole company doesn't seem to fit well. The "GMail" team didn't catch this one.
And I understand that testing is hard and certain things might not testable/doable. But to know how Google try to minimize the impact by doing "something" is far more important.
That's 99.90%, 99.94%, and 99.88% (for the month so far), or simply 99.91% for the entire period.
So, what would you consider "rock-solid"? Personally, I'll echo what was said in the other thread - 99.91% is much better than what I could accomplish on my own, so I'll continue to trust my business to them.
Regardless, 99.9% is really not special or even acceptable from a high dollar host like Heroku. Cheap shared hosts like HostGator can give you 99.9%.
I'm pulling for them, but they've got some work to do.
I may be showing 48^ errors in my per-minute checks but those errors occurred in at least^ nine separate blocks across an eleven hour window.
In other words, we are probably looking at around ten hours of issues, even if not outright outage.
^ I disabled the checks at times because it was just a waste, so the true stats are likely worse.
However, Heroku appears to have no SLA, so its a moot point anyway.
99.5% (or less) = unacceptable 99.9% = average 99.99% = exceptional
0.5% downtime = 216 mins/month (almost an hour a week!) 0.1% downtime = 43.2 mins/month 0.01% = 4.32 mins/month
Getting to that last level is certainly very hard, I hope Heroku will get there, but the service is relatively new, fairly cutting edge, under active development, etc. I would not entirely expect them to reach "four nines", but I hope it is their goal.
These heroku outages are slowly turning into a running gag. I'm not even hosting anything on heroku, yet this is the 3rd time I hear about a multi-hour outage this year. God knows how many outages I didn't hear about.
So, "rock-solid" does indeed sound a bit out of place here.
To me, it seems like they just needed to apply the "Hot patch", instead they panicked(?) and did a lot of unnecessary version control gymnastics, which delayed the bug fix.
If you jump straight to the hotfix, you're basically enlisting the entirety of your userbase to join you in a round of QA, which could be sub-optimal if your hotfix ends up causing some other unintended consequence.
Right?
Absolutely.
However, they rollback should be atomic, which means all pieces of the infrastructure/code should be rolled back to a known-good-state.
When I said "gymnastics", I meant for rolling back one piece of code, only to find incompatibility with other pieces.
I do not intend to judge them, knowing its difficult not to panic in the difficult emergency situation, but not working effect out on paper (or not knowing, which versions of software components are inter-compatible), before actually doing it on code, look pretty novice to me, for a company of heroku's scale.
I really hope this is not an unfair criticism. (handling emergencies are difficult)
1) it take much longer than just rebooting the isolated service. Can you imagine Google shutting down every one of their multi-million boxes, rolling them back to a previous state and spinning them up again?
2) they'd still be at risk for incompatibilities with their databases, etc. the problem with unexpected imcomptabilities is that they're unexpected.
I fail to see, why they need to revert a piece of code, and then realize, OMG... this version of code does not fit well with the rest of architecture, now change it back.
1.) I do not expect Google (or even a small shop, like my place) to revert any piece of code which is not affected.
I, HOWEVER, expect to know what changes I am EXACTLY doing, and what to EXPECT after the changes.
(It should not be black magic, for historical code).
2.) I fail to understand this, why should this the case for older code? I can understand some tricky/edge/minor cases, but whether the architecture/database etc. (major compatibility) is compatible or not, should be possible to calculate BEFORE doing the changes.
I hope I am not over-trivializing the issue, but I still cannot get my head over the approach.
That's not unreasonable. And doesn't sound like "a lot of unnecessary version control gymnastics" to me.
Of course, the problem was that the version they rolled back to was incompatible with the current version of their other services.
Am I missing something here, please let me know?
It's refreshing to see a company that not only acknowledges their outages, but even has a list of all past issues and outages. This transparency can only help them to become better in the future.