Releasing software to the fleet far too quickly broke stuff
rachelbythebay.com
rachelbythebay.com
It seems to me there are lots of opinions on how to change an organization or introduce improvements but ultimately if you aren't making people's lives easier, making them want the change you are trying to bring about, you'll find it almost impossible to succeed, human inertia is just too great.
Usually those things got bogged down in the end because after 1-2 years, the first wave of developers get bored and move on to other opportunities, and the architecture they made is poorly abstracted microservices communicating via undocumented and unstructured REST APIs, with not nearly enough end-to-end tests or traceability to reliably make changes or add features.
[/rant]
Our company has restricted it so CloudFormation is the only way to provision infrastructure because doing things the right way makes our lives significantly harder.
And so the "native cloud" tax continues to accumulate and grow until you spend more and more time and money on just building and maintaining infrastructure...
When I can simply re-run and update the stack without deleting it I do (for example if I just change a property of one of the resources). But I spend a lot of time with stacks that are in broken states for various reasons.
$aws cloudformation deploy. .. —no-execute-changes etc
https://docs.aws.amazon.com/cli/latest/reference/cloudformat...
Correction: When the change was introduced and made it far easier for people to do the thing they are trying to do.
Nobody cares about doing the right thing, employees just do whatever they have to do now then move on to the next task/project. More often than not, changes are making it harder for people to do their job and it's no surprise that it's not welcome.
If you’re even further down the rabbit hole, you’ve adopted devops, and are forcing people that actively hate operations stuff to learn obscure in-house operations tools. That, combined with preventing them from doing the work they signed up for, leads to burn out and resignations.
It can get even worse than that. I routinely see all of the above compounded with infra bikeshedding that delays rollouts by months, and results in weeks of work per team. At that point, the organizational damage ends up being many times the budget of the IT department.
So, yeah, IT needs to figure out how to help people be more productive while simultaneously implementing best practices, and it’s hard. I deeply appreciate people that are good at that (and have worked with some).
Being part of the platform team, I've been in the exact same situation before at an ex employer. We assumed that _infra problems_ were _everyone's problems_ and how can anyone who cares about their application _not_ adopt to the latest-and-greatest-platform (or tool) we released. A lot of time was spent on figuring out the 'best approach' to getting everyone onboard quickly - we tried the carrot and the stick. We failed miserably at both.
The trouble is, as someone else mentioned here - we didn't really go and _talk to the customer_. Internal customers are customers too - and 'product' thinking should be equally applied to all internal platforms. The general rule applies - talk to the customer and understand what they're going through. They have their own sets of problems and priorities.
(a) Is the latest way actually _saving them a lot of time_
(b) Can we take another 2 weeks and make it even easier for them to come on board?
(c) If they're really busy and/or lack the expertise, can we carve out some time to give enough training sessions?
Yet, this never happens and we invariably end up blaming the application teams for _not doing this seemingly simple_ change. It may seem downright stupid from the platform engineer's PoV, but we are partly to blame.
This also leads me to think that the cloud wars will be won by the <cloud-provider> that does this job better - of understanding the customer needs on a day to day basis and building platforms for them.
Somewhat on-topic, instead of re-inventing the wheel on this one, I've recently been using Flagger (https://flagger.app/) and have been exceptionally happy with it. Automated rollback on error rates, metrics or integration tests. It's so nice to finally be able to stop writing this code at every company I work for!
See also poka-yoke: https://en.m.wikipedia.org/wiki/Poka-yoke
(This is another example of how legal-contract drafters can learn from the software world, including concepts such as, for example, orthogonality; modularity; R.O.O.F.; D.R.Y. [Don't Repeat Yourself]; graceful failure; and the benefit of sometimes explaining why things are done the way they are, to help future readers understand.)
Spinnaker invents a lot of new terms and complexities. Flagger just does one thing, and it does it pretty darn well :)
- Any change, whether it's a one-line config update or an entirely new backend, can only go out to a maximum of x% of hosts at a time
- Hosts are taken fully out of rotation before getting an update
- Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions
The first two are easy to enforce programmatically, and the third one relies on the team. While the trade-off is that it's harder to make relatively simple changes, it has served us well so far.
How does that rule work out in practice? If you're doing staged rollouts of new builds because you think there's a decent chance those new builds have bugs in them, shouldn't you realistically expect that rollbacks could have unexpected side-effects?
Expect the unexpected? Sure anything can happen.
But paxys is specifically referring to expected (known) side-effects of rollbacks. For example if Foo and Bar are on versions Foo.3 and Bar.3, and we know rolling back to Bar.2 breaks Foo.3, then you broke backwards compatibility. Generally all combinations of possible rollbacks must continue to work, e.g., {Foo.2, Foo.3} x {Bar.2, Bar.3}. The number of rollbacks is limited to 1 or 2 versions, e.g., Foo.1 and Bar.1 cannot be used anymore.
That other people fall prey to the same issues in many scenarios just makes the pill go down easier for them.
I feel like my "capacity to err over time" graph is U shaped with a bit of an M tendency towards the end - as a newbie, lots of ways to make mistakes without realising, then as my skills increased, realised the "known unknowns", and coded and released very cautiously, but when I blurred into a senior, ran the risk of thinking "I know this" without checking that all my assumptions and prior knowledge held true, and learned the caution of the intermediate developer again.
From the article:
> At that point, it was decided that something easier had to be done. The barrier to entry for safe and sane rollouts had to be lowered even more to bring more teams on board.
Why does something have to be easy for a team of professional software engineers to do it correctly? Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.
In practice it ends up being more like a request. :)
Insisting people use a tool that is capable of doing dangerous things correctly, every time, is white-knuckling. Yelling at them about screwing it up is running your organization by shame. When you do that, eventually only the people with very high pain tolerance ever get promoted, or even heard. Do you really want to work on a team run by masochists/sadists?
Similar to make the change easy, make the easy change, if you want to have a dialog with people about doing better, the easiest way to do that is to give them a reliable tool (literal or figurative), and then start insisting they use it when it becomes obvious they haven't. The people who insist on doing things the hardest way possible can still do it as long as nobody else feels the consequences. But typically what they're going to learn is that where self-reflection is concerned, they're bad at math and they have way more frequent problems and far more collateral damage than they like to tell themselves.
1. I never said everybody is stupid. The vast majority of people in Big Tech are great, but there ARE stupid people there too. I resent the implication of your first paragraph.
2. Who said anything about yelling or other toxic behaviors? There are options other than "engineer a foolproof system" and "institute a sado-masochistic management strategy".
3. I never advocated for doing things the hardest way possible, I advocated for doing things correctly (to the best of your ability) even when it is difficult to do so.
The example given in the article is a team choosing to do a global rollout when the option existed to do a staged rollout. There's myriad technical things you can do to make it easier and faster to do a staged rollout, harder to do a global rollout, reduce the risk of a global rollout, etc. etc. ... but what about instilling a healthy respect for production in your colleagues? That's been conspicuously absent from most postmortems I've sat in on.
So large companies put in place systems to reduce those risks. Unfortunately the default seems to be to make it hard to deliver features.
[0] As mentioned in the article.
Well this kind of bugs me, that explanation can be applied to literally anyone who's ever broken anything by accident. With the benefit of hindsight you can always call them out as ignorant or lazy.
The circumstances in the article were not explicitly lazy or ignorant in my opinion. It was an unusually large update (2 gigabytes), which due to being able to deploy all servers simultaneously, created an outage.
Now, sure they could have done a staged roll-out, but why were they even allowed to deploy to all servers like that? How much due-diligence to they need to do in order to do their job?
Reminds me of [1] that guy who accidentally deleted the entire production database on his first day. Was that "ignorance or laziness"? I would say it's a failure of the company's internal processes and systems.
[1] https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/a...
That’s why investing in tools makes sense: when those people think it’s harder to do things the wrong way then your incentives are aligned. SRE culture around error budgets can help, too: having issues reschedule your project team is a much better schedule than chucking it over the wall to be ops’ problem because people learn that they’re not saving time by cutting corners.
This is basically the core of UX. It's kind of amazing to me that UX is so poorly represented in management tools.
Of course sometimes there's no substitute for learning things the hard way. Where practical, I've found having the responsible team on the hook for cleaning up their mess tends to work wonders in terms of getting people to be appropriately careful. That's not to say mistakes can't still happen, but it cuts down a lot.
To give a theoretical starting point - Acme corporation has a serverless API deployed at api.acme.com powering website acme.com. The staging system is deployed at staging.api.acme.com, and the corresponding web app (e.g. staging.acme.com) targets that API. Staging gets deployed to production via a pull request that triggers CI/CD, which has no ability to do an X% rollout (i.e. the underlying serverless function is updated).
Developer John Smith is writing a feature and merging to his personal branch "jsmith" - he can test his branch with jsmith.staging.api.acme.com and a corresponding development web app. When he's finished the feature, he does a PR to the staging branch, which is packaged into a PR to live.
You may be able to see I'm trying to get input on our own release system :) It's worked quite well so far but I'm still paranoid about not having gradual roll-out - in my experience in bigger companies, there was always some A/B testing in place so bad deploys could be reverted quickly. If the sysop gurus of HN gathering on this thread can pass any comments/critique it would be greatly appreciated.
But you only seem to have one machine in production (or one for the API plus one for the website), so it doesn't make sense to deploy to 10% of the machines.
What you can do is to arrange things so that each individual upgrade can be rolled back. Let's say you want to rename a parameter in the API. You have to make a backward-compatible change to the API so that the existing website keeps working. In other words, you provide two calls, one with the old parameter name and a different call with the new parameter name. Then you deploy that. If there are problems, you can roll back to the previous API version.
The next step is to change the website to invoke the new API call with the new parameter name. If that fails, you can go back to the previous website.
Let's say it succeeds. Now you have a website using the new API with the new parameter name, and an API providing both the old and the new parameter name.
The next step is to update the API to remove the old parameter name. If it turns out you forgot a place in the website where the old parameter is used, you can roll back to the previous API.
If it goes well, you now have a website that uses the new parameter name, and also an API that provides the new parameter name, and the old parameter name is no longer used.
It took a long time, but now it's mission accomplished and problem solved.
The provider hosting your serverless functions might not give you that much control over routing. In that case, you would need to write another serverless function which sits in front of your serverless functions and does the routing. If you ever need to update that, well, then you have to cross your fingers.
I suggest keeping it around for a while. A function not being used should have cost equivalent to a database entry, not a running container or VM, so this should be cheap and the benefits are large. It's still possible to discover you need to roll back a change, even after you've been at 100% for a while. (And no, 'roll forward' is not a sufficient option, especially if that requires coding up a fix you may not even understand yet.)
If new copies of your service don't come up cleanly, you do not power down the old ones. If old servers can't talk to the new service, you get alarm bells going off while most of the old copies are still up and running. You've also partially serialized the deployment so you get some peak shaving.
I wonder sometimes if the problem is that despite our protestations (and things like naming servers by unpronounceable names), we still think of servers running, instead of instances running. So the fact that the 20 copies of my service keep bouncing from machine to machine is at the very least a logistics issue, and possibly unsettling.
The next rant will be: Massive outage because of a bug that was fixed last month. But the fix only shipped to 50% of the hosts, because the deployment system doesn't deploy everywhere by default.
> Some people obviously couldn't be bothered to step their release along over the course of a couple of days, and we had to find some way to protect the production environment from them.
That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
Unfortunately, that approach doesn't scale well. In a larger company, there are always new users discovering the systems and how they work (to varying degrees). If doing the right thing isn't by-and-far the easiest path, then it's highly likely someone will misuse the tool – no matter how smart they are. It's far easier to make the interface really easy to use than it is to educate users on proper usage.
Software engineering turnover rates can be high, like 13%/year [1]. Add to that engineers who internally switch teams. New users is a real concern.
In this case, it sounds like a custom deployment tool, so even engineers with good experience at other companies may not find the feature. Good team on-boarding and documentation is important.
I think code review should catch something like this. Even with the high rate of new users, there should be enough experienced code reviewers to catch the issue.
[1] https://www.techrepublic.com/article/software-had-the-highes...
A common large company problem. People who should be talking are not. New hires are also hired into the "local" subculture of their team and something very, very important elsewhere in the company is not on their radar.
Even if you do all that, you're not going to go back in time and fix things for your customers, and you're definitely not going to prevent the replacement developers from making the exact same mistakes.
It takes really good leadership to build an engineering culture where good practices are valued and people learn from mistakes, which is why filling those positions is so hard and expensive. A company whose solution for every outage is to fire people isn't going to last very long.
>> That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
If people refuse to be careful. If they make a mistake once, teach them. If they make a mistake again, that happens. But if they make a mistake regularly, warn them, then warn them sternly, then fire them.
It is unbelievable how important these technical leaders are, and you don't know how important they are until they're gone.
The solution is for technical managers - at all levels, from team leads up to the CEO - to actually provide leadership to their teams. This can consist of training people about why it's important to do things right and setting up incentives to do things right (e.g., don't promote people who get things done quickly by cutting corners). To me, this story sounds like a classical management failure (been there, done that myself).
(Worse, if the other company has a misuse-resistant deployment system like the one the author alludes to, they might not burn down that company's prod and then beat us in the market while we're busy interviewing replacements!)
You have to fix the system so it's really hard to screw up.
When I was first hired into a manufacturing plant I wondered why it was so hard to get people to make a quality product. It turns out that people are people, and they will try to do the job you tell them to do, with the tools and instruction you give them.
The better the tools are, the less instruction is needed. Output rate and quality is also improved.
The poorer the tools are, more instruction is needed; output rate and quality suffer, and at some point safety as well.
"A poor workman blames their tools"; A poor manager blames their workman after providing them with poor tools.
The very first thing you need to do is assess if the worker is following process and if not why not. If they are not following process, is it because of a mistake or a slip-up? Did they intentionally decide not to follow a process because it was difficult?
Disaster documentaries are quick to point out where workers failed to follow written procedures, but they VERY RARELY point out how common those contraventions are with no negative impact.
If "it's stupid but it works" it may not be stupid, or you may have gotten lucky.