Hacker News Reader

Comment by rantwasp | Hacker News Reader

yeah no. as someone who has worked at Amazon for 7 years I can tell you that it’s not true across the board and there are good “pockets” where there is respect for people’s boundaries.

To add to this, in my time there I worked during the weekend 3 times (i don’t count oncall here). My work day included 8 hours at most. I would get in at around 8AM and leave at 4PM. (optimizing the commute)

I started as an SDE 2 by the time I left I was a Principal SDE.

Do good work, be in a good org.

Amazon is not a huge, homogeneous entity. It’s more like hundreds of small companies that are forced to work together, bound together by senior leadership.

View on HN

"in my time there I worked during the weekend 3 times (i don’t count oncall here)"

Why wouldn't you count it? Do those days not-count somehow?

i don’t count it as in: i worked more than 3 weekends, but not because i had a choice )

I'm trying to interpret your words as something other than "well, yeah, my weekends were interrupted, but it wasn't my choice so it's not worth mentioning." I can't come up with another way to read it, though.

So, don't you think that, in the context of having to give up nights and weekends, you should mention on-call? Your choice or not, the weekends and nights are affected, in fact isn't it kind of worse when it's not your choice?

i agree with you that the oncall is not ideal. it is also widespread in the industry. so what’s the point to complain about something that is an essential part of the job? (advertised or not, if you don’t want to do it you will have to look for another job)

i also think there is a huge difference between being oncall and having to respond in case something breaks vs mandatory death marches all day, every day.

I guess it depends on your definition of widespread. It is relatively common, but it's also far from universal, and there are companies that use on call which have a vastly different experience that what many people at Amazon report. In fact it seems like even different teams inside of Amazon have drastically different experience of on call. Treating it like a fact of life, like the weather, leaves you in a frame of mind that makes it impossible to advocate for change. Your own phrasing "be the change you want to see in the world" could as easily apply to on call as much as it does anything else -- but you won't take that stance if you think it's inescapable.

hmm. no. you do have agency. you have agency during writing the code. you have agency during the code review. you have agency during wiring of alerts (if you do stupid shit you will receive a lot of pages. don’t so stupid shit). you have agency during the events. you also have agency after the events when you do a coe/post mortem. you can defined prioritize things to improve the life quality of other developers.

to give you an example. the team i was in at amazon had 3000 tickets in the queue when i started. anything except sev2s were basically ignored. lower severity tickets would escalate when shit hit the fan. i advocated for fixing classes of issues instead of myopically focusing on one-offs. by the time i left the queue was tens tickets and mostly feature request or higher level investigations.

to give you another example: i would basically remove all alerting that was not actionable. the worst possible thing that you can do is wake up in the middle of the night and not be able to do anything. i would ask for runbooks and the test was “if i take a developer from another team and put them oncall can they function independently 95% of the time”. i would think about what the experience of being oncall was (ie you don’t take people and throw them in the deep end of the pool and wonder why they drown)

so i guess what i’m saying is that oncall for me wasn’t that bad or stressful. it sucks having to be near a computer but I was rarely paged for stuff that broke or needed to be fixed right NOW. (once stabilized our team had 1 sev2 every other week)

Okay, at the risk of putting way too much effort into this, I'm going to assume you have 5 people on your team, therefore you're on call 10 times a year roughly. 5 weeks out of the year then you'd be oncall during a sev2. Assuming sev2's are uniformly distributed you are probably interrupted outside normal working hours ~80% of the time, so call it 4 times a year you have your work-life balance negatively affected by on-call.

you need to generate some 3d graphs and after that write a paper about using gradient descent to improve the WLB of SW developers :)

our team had 7 people and at some point we started sharing the oncall load with a sister team (ie you were oncall for the services on both teams), meaning that you would be oncall roughly once every 3 months. not ideal - but not the worst thing in the world.

I am sure it is not all of Amazon, but that is where the horror stories are coming from. So Amazon as a whole may not have this culture, but the teams with this culture seem to disproportionately be at Amazon.

so, the bigger an org, the higher the likelihood of stories like this popping up.

also, the incentives and their structure play a part in how this unfolds.

It’s also very curious to me that only a certain type of person runs into situations like these: either they don’t know better or they have accepted the conditioning that work is more important than themselves. it’s sad really.

i think it’s really really important to have a good mentor with a strong set of values to help you navigate the corporate landscape.

i also think it’s important to pull a Ghandi (ie be the change you want to see) as often as possible. for example: one think i would inoculate in young devs is that it’s not acceptable to set meetings earlier than 9am or later than 4pm. meetings during lunch were not accepted. i would be pretty vocal about morons that worked over the weekend because they had some free time. i would also push really hard against management that had their sick little practices to extract more work out of the drones. this mostly worked because i got the job done - but again: you have a voice. you are NOT a machine

law of numbers; Amazon Seattle headcount alone is 50K+ people. also, amazon has a service-oriented management model to match the general SoA architecture, where small teams are given massive leeway as to how exactly they run their services. managers and your leadership are thus very make or break.

> I started as an SDE 2 by the time I left I was a Principal SDE.

How many promotions was that? 4? How did you manage to do that in 7 years? That doesn’t sound easy at all.

2 promotions. SDE2 -> SDE3 -> Principal First one was in 2 years. 2nd one took 3.5 years.

Reply on news.ycombinator.com