This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.
The list of managers stating that "they were taking responsibility" and then immediately stepping down was always fairly short.
Wait, what? You think a CEO should step down because their management over-hired a relatively small proportion of employees and had to do some layoffs?
The people laid off and the people not needed were a different set of people, at the time of the layoff.
"Kill one man, and you are a murderer. Kill millions of men, and you are a conqueror"
If you make some idiotic financial decision near the bottom of the management tree, such as... over hiring, you'll likely lose your job or get demoted.
Do it as a CEO, and get a huge bonus.
Some things are cyclical and you need more people for some amount of time, and then you find you need less. It’s not always predictable/seasonal like farming or holiday rush.
Is it wrong for a company to respond to market effects? That there was a layoff isn’t necessarily a sign a company did anything wrong… I think how they actually do the layoff certainly can be done well or poorly.
I’ve forgotten which FAANG it is. But one of them still has more employees than last year even after layoffs. It’s offensive.
It was likely the right call to hire then, just like it might be the right call to reduce headcount now.
Maybe they should step down any time they fail to accurately predict the future?
Frankly - People seem to be forgetting that until 2013, MS was still doing stack ranking and routinely letting go of the bottom 10% of their workforce (and they were hardly the only ones doing it...)
I don't see it as unusual AT ALL that these companies are doing a wave of cuts to headcounts after the large hiring sprees during covid. Especially as interest rates rise, so they're looking to lower debt burdens in the short term and pay off loans made at low interest rates instead of rolling into a higher interest loan in the new environment.
If anything... I'd expect the exact opposite - a CEO that fails to address cost centers as debt becomes more expensive is a liability, and someone the board might be looking to replace (ask to step down).
---
Does that mean I'm not sympathetic to those who've lost jobs? Of course not.
But tech had to rev the engine pretty hard to handle the extra load during covid when everyone was indoors and doing things online, and now that demand has dropped. So they're letting off the gas pedal.
If folks don't like it - blame the game. Work to unionize. Work to incentivize co-ops and shared ownership. Work to increase taxation on these companies and their highest earners (which... if you're in the tech industry almost certainly includes YOU). Don't go work for giant tech conglomerates and then act surprised when they act like giant tech conglomerates...
Of course, once you take advantage of such a representation at scale by deploying tremendously more complex infrastructures, you then have to deal with the dependency network meta challenge lest you inadvertently fall into dependency hell. While towards there lies NP-hard problems, they're still computable to a reasonable degree and I dare say a more robust situation than doing it all by hand like we do today.
The real challenge is the vast majority of devops staff today would really dislike reasoning about such a representation when it blows up in their faces, and I can't blame them for that kind of reaction.
What if multiple sensors fail or it's an ambiguous situation like say you are deciding whether or not to fail over a power circuit and it's a brownout but not a complete power failure? What if there is a systemic problem and it's likely the backup power source is going to brown out too? At some point you need highly skilled individuals, like say trained airline pilots flying a plane who have the authority to override systems immediately without having to jump through hoops.
This is especially true for mission critical systems. Many of the mission critical systems we rely on are NOT built on the cloud, i.e. other people's computers because you want to be really careful about what hardware you are using, precisely how your data center is setup and want to make sure things like a noisy neighbor do not impact you.
Like it or not, these highly trained individuals are going to make mistakes every now and then. A failure like this once every decade or so really isn't so bad. The individual who made this error is likely not a "grunt". I suspect the individual in question will not necessarily suffer any major consequences as a result of this unless it wasn't a mistake but a flagrant disregard for the rules like say bringing a bottle of water into a data center that then spilled or something.
Have you built a mission critical, distributed system that hasn't failed for 10 years? It's a lot harder than it looks. That's how often the NYSE has a problem like this, about once a decade. A lot of things that work in theory, don't work for the edge cases and things that lead to problems once a decade or so are extreme edge cases.
In the grand scheme of things a mucked up opening auction is a minor problem and anyone who did not take the precaution of sending a limit order and sent a market on open order despite it being standard practice to essentially always use limits and go hurt badly will be made whole.
It pretty much boils down to: it depends upon what the business wants to prioritize; operating margin or resiliency. There is an entire subfield investigating the statistical foundations of resiliency, and the general case of N-modular redundancy is in practice implemented as triple modular redundancy in most commercial systems that want to spend in this vector.
> Like it or not, these highly trained individuals are going to make mistakes every now and then.
Absolutely, and here is where the organization's no-blame learning culture swings into action for the well-led teams.
> It's a lot harder than it looks.
We all know this, and we can all help each other get better to deliver ever increasing value to our customers by sharing what works for the context we deployed within!
In real-life outside of a journal article, it's a lot harder than just deciding whether you want to prioritize operating margin or resiliency at 5000 feet.
In real life when these sorts of edge cases happen, you have to understand in minutes or sometimes seconds the tradeoffs in terms of costs to your own company and your customers of one of n specific possible failure modes and risk-manage so you minimize the probability of the catastrophic outcomes. This sometimes may involve increasing the probability of low cost bad outcomes. You can't reason about this stuff before hand. If you could, you would have designed your system to not fail in that manner.
Exactly.
I wonder if any of the people claiming "it's management's process fault!" would be the first to complain about their workplace where they have no autonomy.
There are many kinds of "outcomes". A simple backup would make outages far more rare.
It didn’t.