79 karma · joined January 9, 2022
"And so, a quiet suspicion starts to circulate: where have the senior AWS engineers who've been to this dance before gone? And the answer increasingly is that they've left the building — taking decades of hard-won institutional knowledge about how AWS's systems work at scale right along with them."
...
"AWS has given increasing levels of detail, as is their tradition, when outages strike, and as new information comes to light. Reading through it, one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow. To be clear: I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time."
....
"This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. "
...
"I want to be very clear on one last point. This isn't about the technology being old. It's about the people maintaining it being new. If I had to guess what happens next, the market will forgive AWS this time, but the pattern will continue."
When a company moves from engineering/technical driven to sales/profit/stock price/shareholders satisfaction driven, once it was not possible to cut (technical) corners, now becomes the de facto. If you push the L7s/L8s out of the discussion room, who would definitely stop or veto circular dependencies, and replace with sir-yes-sir people, now you've successfully created short term KPI wins for the lofty chairs but with a burning fuse of catastrophic failures to come.
Amazon is burning and driving away the technical talent and knowledge knowing the vendor lock-in will keep bringing the sweet money. You will see more sales people hoovering around your c-suites and executives, while you will face even worse technical support, that seem not knowing what they are talking about, yet alone to fix the support issue you expect to be fixed easily.
Mark my words, and if you are putting your eggs in one basket, that basket is now too complex and too interdependent, and the people who built and knew those intricacies are driven away with RTOs, move to hubs. Eventually those services; all others (and also aws services themselves) heavily dependent on, might be more fragile than the public knows.
Squeeze the company to the last drop via stock price obsession
If I had to choose between a moose and a rabbit to hunt, I would go for the cute little bunny.
But don't take my choice as guidance, I grew up and lived in city.
Anything limitless/easiest has a higher hidden cost attached.
Why they had to invent "to be the best employer of the world" as a leadership principle?
The more people should stand up and make the stupid policies known, that's the only way this can stop. Otherwise this policies will be the norm for many companies since "it can work"
"This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise."
I think we are hearing more and more managers put people in PIP or other manage-out tools as a retaliation and everyone suggests to apply other jobs when there is a confrontation. Well, when all the companies share the toxic management culture it will be too late ask the culture to change, won't be? I believe more people should stand their ground when they face unfairness, hostility and/or bad management for some more time as they can afford. Making bad managers life easier does not solve the bigger or growing problem. It won't change as long as they go away silently and conveniently.