it's interesting how Amazon is now positioning themselves as a tier 1 employer.
They used to be a tier2.
AFAIK they also have a brutal pager culture where you're awoken in the middle of the night.
They used to be a tier2.
AFAIK they also have a brutal pager culture where you're awoken in the middle of the night.
The other rotation is for one of my dev-team's software, and I'm hardly ever paged - like once or twice a year max. That's how it's supposed to be: by having skin in the game and being directly on the hook for the reliability of what we produce, I'm sure that we automate to a much greater degree than we might otherwise. We have a culture of strong ownership: it's not uncommon to see senior engineers elect to be engaged on all of their team's live issues, even if they aren't on-call themselves.
Some other teams make different trade-offs, and have follow-the-sun rotations or engagements that are more reliant on humans following run-books than automations. There can be a place for that, but it gets old quickly and is not scalable. Good teams will prioritize the work to get out of that above almost anything.
I'm in an on-call rotation (not at AMZ) where I hardly ever get paged, and it's cold comfort. I still have to carry my laptop with me everywhere I go, make sure I have access to the Internet, make sure I don't have one too many at the bar on Saturday night, etc.
For multinational corporations, this has never made sense to me. Why do we insist on relying groggy, "at the bottom of their performance curve" engineers to keep your system up, instead of someone in the middle of their work day?
IMO, if you're waking someone up in the middle of the night, something is horribly wrong, and it's not the issue you woke that person up for.
First, the front-line pager duty is usually hit by an ops eng team first. Each of those teams has an India group and an North America group. If they can't resolve the issue, they page in the developers. Having good Ops Eng guys supporting you is a blessing.
There's also a very strong incentive for teams to write good software and test before deploying when you know damn well that if you're deploying something that doesn't work, you're the guy who will have to deal with it at 3am. Having strong integration and system tests suddenly becomes incredibly important.
Edit: and as dsfyu404ed said, the Ops Eng guys will be smart enough to narrow down where the problem is usually. If you're being paged, it's usually your team's fault.
That's a fairly normal part of Operations work and has been for decades. Most places (I assume Amazon is one of them) will have a pool of lower-tier admins on deck 24/7 who handle and investigate 99% of the issues that crop up. But for that 1% they can't deal with, then they need to be able to reach someone who CAN deal with it.
It's much cheaper to occasionally call an SME every once in a while when they're needed, than it is to hire another 3 or 4 of them to cover a 24 hour roster.
I've been called out in the middle of the night to fix someone else's mess more times than I can count, so I'm 100% behind Amazon's cultural reasons for sticking to this sort of on-call system as well.
Idea being if you're woken up at 4am because of a flaky test or some issue you'll be more inclined to fix it. And if it's a recurring issue, you'll really want to get it sorted or else your life will be hell. It also pushes emphasis on serious testing methodology and the idea that Amazon is a 24/7 business.
It actually makes a fair amount of sense, but it's also brutal and contributes to burnout and attrition. I also think this is starting to change for some teams.
The team I was on (retail-related, not AWS), shifted away from the rolling schedule with your counterparts in India taking over for the other 12 hours in order to push the "promote ownership" BS.
The only problem? Management was constantly pushing for new things to get done on extremely tight deadlines (including "emergency features" that needed to be done and in prod in days when they likely required a week of design efforts to get right, never mind actual dev time) so you have two options:
(1) Develop something stable, with good test coverage and the like, and work to fix it if it breaks... and work 20 hours to get it done.
(2) Shit something out as fast as you can and hope it breaks when someone else is on call (who likely will be too busy triaging SEV3-5's during business hours to even think about spending time fixing the root cause of the SEV1/2, rather than mitigating it and moving on...or, even worse (!) (/s) try to get the actual feature owners to fix it) in order to maintain some semblance of work-life balance.
I'll leave it as an exercise to the reader to guess at which route was generally taken.