A push gone wrong in the name of what, exactly?
rachelbythebay.com
rachelbythebay.com
You can't draw life lessons from some vague description of "something" that happened to "someone". Continuing the rollout could very well have been the best option. Or not. Who knows.
If downtime has consequences then you have every incentive to make rollbacks happen at the touch of a button. From personal experience I know that plenty of teams simply are not at that point for many reasons.
If rollbacks are “hard” in any way for you, then you should know the known consequences of difficult rollbacks—they incentivize you to push forward even when the new release is kind of broken.
For example, let’s say that your team doesn’t test rollbacks and doesn’t have procedures in place. You know, in the corner of your mind, that a version rollout could potentially take your service completely down for days while people hunt down the bug or figure out how to do rollbacks on the fly. This is a predictable scenario to end up in, so at some point I recommend addressing it, or justifying why you don’t.
> Continuing the rollout could very well have been the best option. Or not. Who knows.
Rollback should be the default response in the face of uncertainty when you might have a bad rollout. Ideally. It should be the safe, one-button-press reset back to the way things were before.
Again, I know that’s often not the case. Schema changes are the biggest sticking point, usually. I often see schema changes in two phases—first you update the schema, and then later you roll out functionality which uses the new schema, in a separate rollout. This is tricky, but most updates are not schema changes.
Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things need to be. As the team evolves and the product matures, the release tooling gets better. You read articles like this and you can make the tooling better without the object lesson. No one-size-fits-all solution, obviously, but… painless, one-button rollbacks are one-size-fits-most for sure!
Someday, someone writes an integration test that sets up a new and old version and makes them talk to each other, checking that they both work correctly.
Another day, somebody writes an integration test that does a rollout from the previous version, followed by a rollback.
And someone else adds more monitoring signals so if they see service failures above some threshold it automatically rolls back.
These are all evolutionary steps. Yes, they don’t apply to all services. But—most of the services I run are somehow making money for the company, or people depend on them for critical pieces of functionality related to the company’s bottom line.
Not all rollbacks are the same. Even in a theoretical scenario where you can roll back the code to a previous version with a push of a button, it's not always possible due to many factors.
Sometimes new code creates and persists state that the old code can't handle and rolling back would just make the old version blow up because of it. Sometimes clients can't talk to the old code after talking to new code and expecting new protocol.
The point is, there's a lot of factors that could make one or another remediation strategy more or less viable, and sometimes rolling forward is the best way to proceed.
The article tries to advocate that rolling back is ALWAYS the correct thing to do, which is demonstrably not true for certain circumstances.
Or worse, the new feature involved the tracking of something in the real world, such as inventory or money, and rolling back after the feature had been used by a real customer would lose that.
The number of times I’ve had to explain to webdevs that when a customer asks where their money is “in a temp table” is not an acceptable answer.
These are both severe bugs that should have blocked deployment of the old version. Software should gracefully ignore state or data file content it doesn’t expect—not blow up. Well known techniques like fuzzing can help catch this kind of defect. Clients should be able to probe which version they are talking to and negotiate the protocol. This is kind of basic versioning cleanliness.
One advantage of 'learning as you go' is that oftentimes you can afford to be messy and cut some corners when you're just starting out without it impacting people too much.
I don't think anyone would disagree that having a rollout process that can be easily rolled back, utilizes things like feature flags, incremental rollout, and care around schema changes is a better process, but it's undeniably slower than "just roll it out and deal with the crashes", particularly when there's not plug and play standards for a lot of those processes and you need to build them yourself.
And if you have a tiny amount of users who are the early adopter type, messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster.
Unfortunately, my personal experience with software engineers pushing to “roll forward” on failure is that they want to meet arbitrary deadlines. Metrics that are good for the SWE (e.g. if we roll forward, I complete my Q4 OKRs) but not good for the team—total velocity isn’t negatively impacted by a single rollback, at least not much, but reliability is positively impacted by rollback policy.
Toggle the flag. Feature is disabled.
Obviously if you know in advance you should not even start, but if you notice after you are halfway done? Meh, chances are that a rollback would take as long as finishing the push, and either way the service disruption continues until it’s finished.
Assuming every machine sends data to all its peers regularly, if I have a cluster of 10 A machines and upgrade 1 machine to B, and the A machines crash, that means I lose 90% of my capacity - likely making the sole survivor get overwhelmed too.
"Outages are OK, clients can always reconnect or try again later" isn't the usual standard HA databases are operated to :D
But I guess we just have too little information to make a really informed decision about what you should do.
A is “baked in”, so it should be the default.
There are simply too many classes of bugs which take time to manifest. Memory leaks, cache staleness, etc. So, for very good reasons, you consider version A preferable to version B.
Oh, I thought of an ancient copy of Windows for Workgroups taking down the Windows NT 4 domain servers every single day. Real story. But that case was version A crashing version B, not the other way around.
Obviously, avoiding deploying broken stuff is the best case, but preaching about fire prevention in the middle of a firefight is quite a bit too late and not terribly helpful.
The best option when a defect in B was detected would have been to rollback so that you aren’t forced into a scenario where every fix has to be a fix forward.
Why would you make that assumption?
If someone is paying for a service, you are obligated to provide that service. It may be nothing more than another (for example) random point-of-sale system to you, but to the business on the other end, it could be fatal.
Sure, there are scenarios where you can justify rolling forward when you're spraying packets of death everywhere. None of those scenarios are the default - the default is roll back. (Imagine if there was some latent bug in the new release corrupting data in transit that you didn't notice until 100%, and you're not sure where it is, just that it's happening - now you literally can't roll back, because your current release will hose the previous one. At this point all you can do is go ¯\_(ツ)_/¯.)
Also, the title of the post screams pretty loudly to me that the people making the call to push the backwards-incompatible release didn't seriously think through the implications of their decision.
It’s generally very risky to put your state of world into a place where the old code can’t even look at it. Because rollouts and fallbacks become much harder to orchestrate. Mostly it provides the property where you only need to worry about writing environmental upgrades and never downgrades even if you sometimes fall back to older builds.
However the story in TFA is a bit different. What we have is an incompatibility between two versions running side-by-side. Nothing wrong with Version A or Version B per say, but just that you cannot roll out B progressively. Obviously the only fix is to immediately get your environment homogenized in A or B, and maybe there is actually a valid decision / risk analysis to go one way or another. For example, if you don’t respect the backward compatibility edict outlined above, it could be tricky to downgrade your B instances back to A depending on what state changes in the data B may have already made.
But the real problem is actually one level deeper. The fact that B got to the point of being rolled out with a catastrophic incompatibility with A means that DEV is fundamentally not testing for the ability to do a progressive rollout before they actually start doing one. The new B code was never actually tested in a heterogenous environment with A, meaning these rollouts must have been happening on blind faith alone.
The purpose of partial rollouts is to observe if there are bugs. The unusual case here is that version B's bug is forgetting backward compatibility, causing it to rapidly take down version A too. This means you can't simply rollback B as usual. After you rollback you still need to fix broken A instances. It's a lot of work but it would be the right thing to do.
Instead, they went all in on version B to avoid the bug they just introduced. This is completely reckless. You're skipping your usual process so you never get a chance to observe for other bugs in B. You should actually expect B to have even more bugs in it given you know it already has one major production breaking bug.
Going all in on a version that you're not confident in just to fix the one bug you know about is stupid.
In that world, "fixing" B could involve sending invalid or unintended data to work around the problem in A, or patching A before rolling out B (which, when you're at the point of rolling out A.2, you may as well just roll out B)
It sounds like A was buggy (it dies when it receives a particular sort of request that it doesn't understand).
Maybe the devs knew about the bug in A? Maybe they decided it was better to make version B work as intended rather than try to throw in a one time hack to be backwards compatible with a broken version that was going to be a thing of the past in... How long does deployment take? Is it even long enough to matter? Maybe the most efficient solution was to allow some of the instances to die during deployment? What kind of application is it? Possibly a few failed requests while you wait for B to fully deploy isn't a big deal. Maybe the client auto-retries quickly enough that the dropped connection is negligible. Maybe there isn't a client at all.
The possibilities are pretty much endless given what we know.
And while if age with the practice of roll back first ask questions later the fatalistic tone does not match the documented impact.
And of course it shouldnt happen in the first place but hey shit does happen. No one has ever managed to figure that out, and testing isnt a silver bullet.
If you are seeing failures and want to roll forward, you should be able to clearly articulate why this is better than rolling back, and what makes this particular scenario different.
Otherwise, I’m going to be on the side of the armchair observers calling for rollbacks.
Better yet: build code and processes appropriate for long-lived massively distributed systems that will be incrementally upgraded over time. If the system is architected right, it will never get into a state where a rollback recovery becomes necessary. This is why we have Content Negotiation. This is why we have Erlang. This is not a new challenge by decades, and there is a huge body of expert knowledge and tools upon which to draw when implementing such systems, so any such complete catastrophic basic failures now are entirely down to PEBKAC, and remedied by a swift clue-by-four with a pink slip nailed to the end.
There is a very simple principle underpinning distributed communication: servers should never make assumptions about who their clients are and what they need. Talk to the client, find out what format(s) it’s willing/able to accept, and serve it the best match. A client should never need to know, nor care, if it’s talking to a version A server or a version B server: if the client says “I only understand version A data” then it’s the server’s job to serve up data in that exact format, not to pique and whine about how old and out of date the client is, push it version B data instead, and then blame the client for choking on it.
Indolent developers who approach IPC the same as local messaging and then blame everything but themselves when it barfs all over the place are the absolute bane of this industry, and this shit is entirely on them. And shame on the equally inept management culture that continues to let such incompetent amateurs get away with it.
There will be bad rollouts. I know of no set of practices which prevent bad rollouts. You talk about “indolent web developers”, well, that’s not productive and pointing fingers doesn’t make your software work. Your software will, in spite of your best practices, in spite of hiring the best people, in spite of experience, sometimes fall over.
Yes, it will sometimes segfault.
And yeah, I count on the fingers of one hand the number of web developers I’ve dealt with over the last decade who I’d be willing to cross the road to piss on were they on fire, and still have fingers to spare. They’re just the worst of the worst.
There was NO excuse for the failure described in the article. There was NO excuse for the described response to that failure. Yet such base incompetence and gross irresponsibility is not only systemic but entrenched, rationalized, and embraced in this industry. With responses like yours, it’s not hard to tell why. Buncha Children.
In this case, right. In general, stuff happens. There’s a tradeoff between reliability and effort. The correct reliability target is not 100%, because you can’t get 100% anyway, and as you approach 100% reliability the cost increases without bound.
I’m not sure what the rest of your comment is about besides taking a big shit on web developers and talking about how awful they are.
There is a precious small percentage of developers who are really good at making reliable systems and they have the burden / responsibility of spreading their knowledge. They work with the other actual developers you hire, those beautiful imperfect developers who cut corners, test in production, and don’t write tests.
You make changes to your culture and your practices. You build monitoring and rollout automation. You increase test coverage.
If you just call people children you’re going to be there, on the sidelines, watching other people build real products. You don’t teach people by making fun of them.
So many words to non precisely say one to three (assuming a five finger hand).
Plus, we don't know if the bug resides with A or B. Maybe B is triggering a previous unknown bug in A instead of B being buggy.
You cannot make the decision to roll forward on the basis that you guess that rolling forward to B is stable. You don’t have any evidence for that—just a guess. However, there is solid evidence that rolling backward to A is stable—it has been running that way for a while.
You must make the decision with imperfect knowledge. The correct answer is to roll back.
You can rightly argue that client A was irresponsible in not protecting itself more intelligently against invalid or unexpected inputs (dumping core is the crudest, bluntest protection there is; not the best choice in a distributed environment where transmission errors are a fact of life); but the system as a whole worked.
Had server B continued to supply client A with data in the format that client A expected to receive, no crash would have occurred and the entire rolling upgrade would have gone without a hitch. But no, the lazy irresponsible corner-cutting assholes couldn’t be arsed doing that; they just start pushing the new data to everyone, and then blame everyone but themselves when that goes sideways.
“The correct answer is to roll back.”
The correct answer is never to get into a state where rollback becomes necessary. Though having failed to do that, and so ended up in exactly this state, immediate rollback of B to A may well have been the next-best response, followed by system audit to determine what integrity/data loss has occurred and post-mortem of the procedures used, and subsequent corrections so that it doesn’t happen again.
But if you think a bunch of cowboys who were only too happy to shirk their responsibilities during the (private) development phase are suddenly going to own up and accept personal liability when it blows up in the (public) rollout phase, then boy, do I have an eight-figure Enterprise-y grade bridge to sell you.
We occasionally roll out bad software. I know of no reasonable set of practices which can avoid it.
I honestly don’t understand how you would expect to make this possible without an obscene budget + insanely slow pace of development.
Honestly, as a profession we very rarely roll out anything else. It’s one of the reasons we should be designing systems and procedures that are fault-tolerant from the very start.
In this particular case study, a very small, simple, obvious, cheap step (enabling server B to talk to clients A as well as clients B) was not taken during design and development, nor caught addressed during testing, resulting in very large, complicated, costly failure which the culprits then tried to mask instead of owning it like the professionals they’re supposed to be.
That so many other techies should automatically switch into CYA mode when it’s not even their own fuckup that we’re talking about here is a damning indictment of modern developer culture’s attitude toward professional responsibility and personal liability.
It’s excuses all the way down.
Not a place from which you can reasonably judge unless judgement is your only goal.
When dealing with a complex system, roll back first, then figure out why it broke and how to fix it for the next attempt.
They have burnt their bridges by making it impossible to revert to older version. In effect, if someone can not run a newer version, they will be forced to completely abandon the software.
Of course, that still counts as "meeting needs of the business" if the only thing your business needs is surviving for a couple more days. But that can be accomplished a lot more easily by doing nothing — your business is unlikely to vanish in puff of smoke tomorrow, as long as you simply sit on your hands.
IMO, they could have halted distribution of buggy newer version, released minor fix for older one to stop it from crashing, then finally a backwards-compatible major release. That's more or less same effort, but with less damage to reputation and users of software.
My guess is there is some detail missing that clears up why what happened was wrong. Sounds like A had a bug that rose to the level of being a security problem of the denial of service variety, so redeploying it seems like a terrible idea.
People generally keep using old versions of software for years. 90% of users don't care about new features or performance improvements or bug fixes that don't directly affect them.
So I understand what they did. If they indeed introduced a version C that had a workaround to avoid crashing A, then they would have never gotten rid of A. No matter how hard they pushed C, some people would just continue to use A.
But if A starts crashing, and there's an update available that fixes the issue, that's a really good motivator to get people to upgrade.
Sure, it'll suck for everyone for a short while, but it'll soon be over once everyone has upgraded.
And, as a dev, I also know that crashes are not the most serious kinds of bugs. They are usually easy to diagnose, and people realise that something is wrong right away -- much better than a bug that silently corrupts data, or a bug that causes subtly incorrect behavior.
Not having the backbone to make that decision and allowing the call to forced at some arbitrary time by a bug and concluding the bug was a good thing (or at least had a silver lining) because it allowed the org to move away from A suggests a possible leadership decisiveness problem to me.
Developers of one service simply don't have the power to force other developer in another part of the organization to do anything. That's assuming they're even aware of the existence of that service and who is supposed to own it.
Having the decision "made for you" and being happy/relieved about it doesn't seem like the healthiest or firmest hand on the tiller way to run a company.
the team I'm working on has successfully turned off ~10 systems with/without replacements, and got all of our clients within a much larger than 500 person org to migrate fully.
Fact is, it's a nightmware to handle. It just takes one client who can't or won't upgrade and it's blocked forever.
You definitely want to make sure that updates are backwards compatible. We use a mesh and once there was an incompatibility on the mesh protocol. You want to make sure this will never cause trouble. We decided to use another channel all together.
Our next update will update the bootloader and radio stack as well. Scary! This is an update that uses multiple intermediate bootloaders so we fall back to a working system at each step. We definitely do not want to have bricked devices out there. https://github.com/naveenspace7/bluenet-bootloader/blob/inte...
We almost didn't find a bug nevertheless. In our older devices there were some programmed with a binary generated by objcopy without the --gap-fill 0xFF flag. It lead to an hardfault in initialization of the new file system. Brrr. https://github.com/crownstone/bluenet/issues/73
Except this time they did seem to dig their way out of the hole :D
In particular, I don’t like when folks write blog posts like that instead of talking to their colleagues. I remember being on a receiving end of such post as an ‘anonymous moron from someone’s work’ and as you can imagine, there was infinitely more nuance to the matter than my colleague ever cared to mention.
I would counter the “don’t rush” bromide with “anything worth doing is worth doing adequately.”
https://github.com/bitcoin/bips/blob/master/bip-0050.mediawi...
In that case they decided to rollback.
Of course well-thought release policy will help immensely (how to test, when to release, staged rollouts, etc...)
because it gets really hard to reason about the state of mutable resources like databases and storage that is being written to by two different versions of the application simultaneously.
in this situation pushing on with the deploy or rolling it all back are valid options.
however everyone knows rollback are always more difficult than deploys
And even if there aren't any other bugs, this scenario must have caused a lot of stress for the people in charge of the rollout.
The tale they've shared should be obvious: When a deploy is causing problems, stop. And here are some generic techniques that could have been used to avoid ending up in a situation where one might choose to plow ahead instead of rolling back.
How is this controversial to anyone? This is Formalized Change Process 101.
I have the opposite impression that HN collectively appreciate Rachel posts very much, judging by the sheer number of her blog's posts that are upvoted to front page recurrently. Not many personal blogs reach that number of upvotes in HN.
*Maybe even all kind of posts, but the ones that are technical and about software are just the sweet spot because HN is full of software engineers.
https://news.ycombinator.com/item?id=9224
"Rossbythebay" is good though.
Yes, yes, many things, certainly. Obviously, fix-forward is very bad because of these many things.
As a highly compensated software engineer, of course I know what these things are, but, just checking, what would you say they are?