Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.
I've been an SRE on a tier 1 GCP product for over three years this is not the case. In my experience, our systems have only gotten more reliable and easier to understand since I joined.
It's not like there are only a few key knowledge holders that single handedly prevent outages from happening. In reality, you don't need to know shit about how a system works internally to prevent outages if things are setup correctly.
In theory, my dog should be able to sit on my laptop and submit a prod breaking change without any fear that it will make it to prod and damage the system because automated testing/canary should catch it and, if it does make it to prod, we should be able to detect and mitigate it before the change effects more users using probes or whitebox monitoring.
This is happens for 99.9% of potential issues and is completely invisible to users. However it's what's not caught (the remaining 0.01%) that actually matters.
Google doesn’t have nearly as hard a time retaining good people as Amazon does.
Strict adherence to the "hiring bar" means we fail to bring in good people who aren't desperate enough to act out the cultish LP dance during their interview. Hiring new grads seems to be the only area where growth is not stalling - but that can obviously only help so much.
My team is hiring for 2-3 people and we are being buried alive without that growth happening sooner - but I can't in good conscience recommend this place to anyone I respect or like.
What is the "cultish LP dance" here that is weeding good people out?
>"My team is hiring for 2-3 people and we are being buried alive without that growth happening sooner - but I can't in good conscience recommend this place to anyone I respect or like."
I appreciate your candor. Are you in a dev role or are you on the SRE side? Is your description true across pretty much all teams/services then?
The "culture fit" interview process focuses on leadership principles, so lots of questions like " tell me about a time when you went above and beyond for a customer". Being yourself will get you nowhere, you need to research the questions and the script that is expected of you.
> What service does your team work on?
I'm a partner-focused SA, so not a developer and not aligned to a particular service.
No offer, recruiter emphasized that they were halving the “cool off” period for me (so I could interview again soon), and maybe they do this for everyone, but it’s clear there was one interview making the difference here. Interesting that this is apparently a common problem.
So, when I used the example of designing and starting to build the replacement e-mail system for ASML for literally zero additional cost [0], I pointed them at the URL for the Invited Talk that I gave based on that work. And when I used the example of when I broke all e-mail across the entire Internet, I pointed them at the URL for the article that was published in The Register. When I talked about what I had learned about Chef and DevOps, I pointed them at the invited talk I gave in Edinburgh entitled "From zero to cloud in 90 days" and the accompanying tutorial I taught called "Just enough Ruby for Chef". And so on.
I really feel that having the URLs to backstop my stories helped me sail through that part of the interview.
In my case, I wasn't being hired as an SDE, so there wasn't much programming tests they wanted me to take -- one of their senior developers did connect me to a shared coding platform, which we really just used as a shared whiteboard. He asked me some questions on how I would solve some problems, and I used my 30+ years of experience with Bourne shell and bash to show him stupid first order solutions to those problems and then we talked about what some of the second and third order solutions might be.
The longer I work at AWS, the more convinced I am that everything depends on the people you're working with. There are good teams and bad teams. There are good managers and bad managers. And if you can find a good team with good managers, then you're golden.
In this respect, I don't think AWS is materially different from any other employer I've known.
[0] They had already bought all the hardware, including some stuff I scavenged from a closet where they had been sitting new-in-the-box for a couple of years; the OS was covered under their site license; all the application software was open source; and my time was free because I was already there under another contract)
What gets me is that every recruiter who reaches out (hundreds by now I estimate) wants me to complete their timed coding test to qualify to interview again.
I found Google’s interview process comparatively much more respectful (although more demanding), and have been happy working there instead.
I’m sure it’s helpful to weed out applicants who actually cannot code, but what’s the point in doing it twice?
Thank you!
Speaking of the research, the recruiters email you the principals and specifically mention to you to review them and to consider them during the interview process. They even send you a document about the STAR method of interviewing to help you have a smoother interview. To me as a guy from Baltimore who doesn't know anything half the people here do. I don't think the interview process could have been smoother.
Is "SA" systems admin here?
Broadly though it is a pre-sales role to help people get started, followed by ongoing guidance as the customer iterates (this is the part which often turns into free support).
In my experience, they also give the toughest programming questions. It is a lot of prep overall.
Now I work at one of the slightly more sane FANMAG (to include $ms) companies.
Pretty sure I dodged a bullet, maybe the engineering manager spared me because he liked me more than I realized.
Ones I'd never consider, even for a cushy VP role- FB, Goggle.
Ymmv.
Maybe the service teams are less heavy on LPs due to not being customer-facing?
Sounds like regular skeleton-crew enterprise IT.
"What were your duties at your last position?" "Performing the daily ministrations and singing the praise of the machine god."
Honestly the lore of w40k is quite fun to read, if you’re into dystopian and fantasy sci-fi.
It seems like both sides are fine to have them be reconciled, but it's an important narrative gadget that can be used to get humanity to fight itself in-universe.
Also interesting is that humanity's "lost" technological progress seems to eclipse some of the other races in the W40K narrative, with even the Tau (space dwarves with robots) and Eldar (space elves with crystals) freaking out when humanity brings giant robots, because the sheer physical impracticality of a gigantic human shaped robot is noted, with nobody aware of how they continue to work.
BattleTech had something similar: it was considered cheaper to keep replacing humans than to replace the mechs, because hardly anyone knew how to repair or build mechs.
Scientists only get to talk to the public every 100 years or something, wasn’t it?
IT was never allowed to talk to scientists.
Seemed like a modernist idea, even at the time of publishing.
The Great Resignation had to have taken a huge toll on regular enterprises. There are probably going to be some unlucky (or lucky, depending on how hardcore they are) people in the position of maintaining aging legacy systems and retrofitting them into the future.
COBOL, for example, is becoming a lucrative language for people in the financial and insurance industries. Legacy Java is all over the place, I’m sure. Legacy .NET is in the middle of a huge industry retrofit, (.NET 5 was the official post-legacy rebrand and they’re on to .NET 6+ now).
The Great Resignation was people leaving jobs they didn't want (front of house/service industry/gig) for jobs they did want (career-track jobs) after those jobs resumed hiring again after the pandemic settled down.
Labor force participation went up not down due to 'the great resignation.'
What's causing people to believe that the latest round of attrition is any different?
I’d definitely agree that it is probably harder to become good in older organizations - the technologies are probably generations behind the current state of the art and the learning curves are high for those older technologies.
Just thinking through keyboard, but it’s probably reaching the point where enterprises need to evaluate aqui-hire or outsourcing entire development departments due to attrition, due to the incentives to leave for regular employees.
If your company is hostile to people sticking around for decades, then it makes it that much less likely that you end up stuck with an machine that relies heavily on poorly documented tribal knowledge that's likely to start falling apart as your core people start cashing out.
The Great Recession
The Great Resignation
The Great Dying (due to COVID-19)
Repeating this wise comment: https://news.ycombinator.com/item?id=23769427
The COVID death counts are hopelessly over-counted. This is why there's a cottage industry of people pointing out things like "COVID deaths" which mysteriously also suffered from being murdered, or drug overdoses, or undiagnosed leukaemia.
Then you get into the problem of care homes being authorised to report COVID deaths without any testing or formally trained opinion at all.
https://www.medicalnewstoday.com/articles/how-are-covid-19-d...
Can you in good conscience say that the typical rate of death remained steady while the following happened:
* a majority of given populations remained at home (lock downs - meaning no travelling to work in multi-tonne death traps)
* practised increased hygiene protocols (masks, more frequent hand washing)
* did not visit elderly homes (at least, less than usual)
* many people were reluctant to get timely health care (due to fear of catching COVID from medical facilities)
* ate worse and excersied less
* infected elderly patients were sent back to their nursing homes (to typically infect the entire facility)
There are so many confounding factors that on the face of it, should result in a radically different death profile... and almost every country faced the above to different extents.
Anyone claiming to be able to work out the actual number of people who died from COVID-19 from excess deaths is being disingenous at best.
I think you have that backwards: disingenuous at worst, a scientist at best.
There’s very likely to be severe undercounting of COVID-19 due to the same reasons that crimes of victimization are underreported: shame.
Not to mention the swaths of homeless and disabled people that probably didn’t get counted.
Of our senior engineers & team leads, 70% have joined in last 6-9 months.
Only 3 full time senior engineers with 2 years or more tenure.
We've grown during COVID but we've also just burned through people.
Turnover has hit the point where we stopped doing going away zoom toasts.. people just sort of disappeared.
I would, and have. Granted not with an entire org, but it’s still good form to take a few moments to say goodbye.
If we can have meeting after meeting for working groups, agile kayfabe, status reports, etc for hours recurring weekly.. we can spend 15min on the phone saying thanks, good luck, and see you again a handful of times per year when a teammate leaves.
That's the dream. Obviously there are companies that sink between v1 and v2, but that's life.
Fundamentally I think the cloud business is robust, it's a fundamentally reasonable way of organising things (for enough people), which is why it attracts customers despite being arguably more expensive.
I've been in this situation in much smaller scales, and yes, you'll see massive drop in productivity but that's the cost of going from prototype to product.
We're slowly but surely converting the world's institutional technical knowledge into re-usable and automated runbooks.