Working at Amazon | Hacker News Reader

Working at Amazon

tbray.org

82 pointskawera60 comments

The biggest flaw in working for all those mega corps, is that despite their size, they still haven't fully embraced remote work. Yes there are island, but Amazon are still looking for that "2nd city" to locate in. Some people may dig going to an office, I don't. If you're lucky to be early in a given neighborhood, you can probably retire on renting out your house. If you're late, like applying to Amazon now would be, you're just a blue collar worker, with higher turnover - you earn more and spend on rent. Especially in places like Seattle if I was to relocate with the family, I'd be an indentured servant, with obligations to pay high rent/mortgage, I'd probably have nowhere to change jobs to, if Amazon didn't work out.

One can dwell on small things like which version control does a company use, but ultimately, they are failing to deliver on the "quality of life" thing.

> Especially in places like Seattle if I was to relocate with the family, I'd be an indentured servant, with obligations to pay high rent/mortgage, I'd probably have nowhere to change jobs to, if Amazon didn't work out.

That's quite the opposite. Wouldn't living in a big city mean more choices? Seattle for instance has high paying job options from Microsoft, Google, FB if you decide to leave Amazon.

However, yes having more acceptance of remote work would be great.

Aren't the Google and FB offices in Seattle relatively small, satellite offices?

Yes. The optionality relative to rural america is excellent. The optionality relative to the bay area is very poor.

I don't know about Google but I interviewed at FB Seattle and was told it's the second largest FB office after Menlo Park, with thousands of engineers and plans to keep growing.

I don't know about FB other than the building looks big, but the Google Seattle presence is certainly not small and is growing:

https://www.seattletimes.com/business/technology/google-plan...

Google Seattle-Kirkland is the third largest Google office. I wouldn't classify it as "satellite".

There is no such thing as "Google Seattle-Kirkland", no matter what the managers and recruiters may tell you. There is "Google Seattle", which is located in the Fremont neighborhood, a small and friendly office which I would definitely classify as a "satellite"; then there is "Google Kirkland", located on the other side of a large lake, over in the direction of Microsoft, which is certainly a big place. Still... Google is all about the mothership. Everything is centered culturally on Mountain View (called "MTV" internally).

Well... that's how it was when I worked there; things could have changed a lot in the last five years, perhaps. It sounds like the new Seattle office they're planning to build will be less of a satellite.

What size would be "relatively small" in your measurement? They're certainly smaller than their Bay Area headquarters, so "relatively", they're smaller. But for FB, at least, there are many Seattle-primary teams (ie, teams where all or most members are in Seattle), especially in infrastructure and storage.

I've worked over two years in each of the FB Menlo Park and Seattle offices, and since the move to Seattle, I haven't missed anything from the headquarters (especially once the BBQ place opened).

Wouldn't living in a big city mean more choices?

I guess the OP is only willing to work for one of the big-4?

Here in DC, between private sector software (where I work), public sector and defense, and consulting/contracting, there are tons of options all across the spectrum of job roles and salaries. And these are located all around the area, not just in a single part of the metro.

> I'd probably have nowhere to change jobs to, if Amazon didn't work out.

Seattle rents/homes are, compared to SWE salaries, not that high (Yet), and Microsoft, Facebook, Google, Tableau are all hiring.

at senior developer at a major company like amazon we're talking...what? At least $200-250k/year? Even in Seattle that doesn't trap you in working-poor conditions.

I am sort of a remote developer for Amazon.

I'm a 'deployment engineer' who travels most weeks to different fulfillment centers (warehouses) for work, and when I'm not traveling I'm usually writing code for my team.

Of course, by saying that I reveal the real flaw in my entire remote role: aren't I supposed to be working instead of posting comments on HN?

I hardly think that is unique to remote work, as evidenced in the same manner by my comment from my office.

I go into work every day. I wish I could work remotely. But I don't, because I find I hate every time I'm forced to interact with the people who do.

Working remotely makes you a pain in the ass to interact with.

You may not be working at a place where remote work is part of the culture. When everyone is remote, the tools and processes are built around it and it no longer feels like a “pain in the ass”.

In my experience, even in those places (I've worked in two so far), it's still a pain in the ass.

Exactly - in my organization, 9/10 times coworkers from the same floor of the same building join the zoom/webex rather than walk to the meeting room.

In that culture, being home 2 days out of 5 is no problem. Being 100% remote (like some of our folks) seems to work out fine (though I will admit being 100% remote as a manager is not as feasible).

We all wear headsets and despite a somewhat "open" office layout, it manages to work out decently (thru proper usage of mute and pausing screen).

Half my team is remote, including my lead test engineer, product owner, and UI developer. My manager works remote 2-3 days per week. None are a pain in the ass to interact with.

Remote work can work, but it takes effort, both on the part of the employees (to be available, have good internet connections, etc) AND the employer (to provide quality tooks to interact with remotes).

Amazon has one thing going for it that I think is a huge deal for a standard BigCo job: if you don't like your initial team, you can switch ANY TIME. And if you're an engineer, you can probably join any team you want.

The reason this is such a big deal is that when applying to any company, you'll get placed on a team and it's really hard to get enough information on the culture, mission, management, etc. Once you're inside? Tons of information, and a lot of coworkers you can ask. This really reduces the downside risk of a new job.

I know this because I work at Amazon, and switched teams very early. I'm much happier on my second team, and probably would have quit on the first team.

What was wrong with the first team? How large are teams, generally? Are we talking like, six people, or sixty?

I'd also like to hear more on this, if possible

Teams are generally 4 to 8 people. At Amazon you frequently hear the term "two-pizza team", meaning the team can be fed with two large pizzas. That's the size they usually target.

I didn't like the project I was assigned, didn't like the underlying design of the org's charter, didn't like the hours. I also got transferred between managers. I didn't really like it from the director down. It sounded way cooler from the pitch the hiring manager gave me. Team size is not a well-defined concept, TBH -- hierarchies can be structured in very different ways, and a team can be comprised of people with different managers. Also, if you love your director and adjacent teams, it would be unlikely you'd hate your own team.

Once I was inside, I found a few teams that sounded cool. At that point, I could easily meet people on those teams, read survey results on the managers, ask around about the new teams, visit the offices at 6-7 PM and see who was still around. This is somewhere between 10-100x the information I could get as an internal employee than as an external applicant. This led me to joining a team where I'm very happy. And if things went really badly -- I could have switched again! (Might look bad if you switch a ton though)

At FB in engineering, you generally don't get placed on a team, you choose which team you join, only after 6 weeks of "Bootcamp" where you ramp up on the technical and cultural environment. That said, you're right that even with that, people can end up joining teams which are a poor fit, so internal mobility (switching teams) is definitely valuable here too.

I think AWS is probably the place to be Amazon. The whole idea of the cloud is that everything gets automated so good and clean engineering will be rewarded. I think working on the Amazon e-commerce is much less fun because it's much messier to deal with issues like customer support, taxes and other stuff.

it's interesting how Amazon is now positioning themselves as a tier 1 employer.

They used to be a tier2.

AFAIK they also have a brutal pager culture where you're awoken in the middle of the night.

Isn't that the point of a pager system?

Many companies have overseas teams that take the night shift, eg: every on call dev has 12 on during the day, 12 off at night. Amazon does not based on cultural reasons (allegedly) because it promotes ownership and accountability for those teams.

Idea being if you're woken up at 4am because of a flaky test or some issue you'll be more inclined to fix it. And if it's a recurring issue, you'll really want to get it sorted or else your life will be hell. It also pushes emphasis on serious testing methodology and the idea that Amazon is a 24/7 business.

It actually makes a fair amount of sense, but it's also brutal and contributes to burnout and attrition. I also think this is starting to change for some teams.

Amazon encourages developers to be on call, but there are also rolling on-call systems so no one gets paged in the middle of the night. A team I work closely with does this. It isn't very stressful for me, I'm just on call for small, well-defined intervals, something like 1 week out of every 6, and only high severity issues will cause a page. In the last year or so I've been on call I was only paged after normal business hours once. Of course, Amazon is an absolutely massive company, so different teams may have wildly different experiences.

I think it's different for each team, especially the AWS groups. I know some support engineers that do 12on/12off rolling calls like you said, but others where it's one person on call, 24/7 for a week and they catch everything, not just SEV 1. It really just depends on your org/boss but the AWS org is known to be particularly pager-slaved.

Meh, I really don't buy that reasoning - but maybe it's my current job where 9/10 "got woken up at night" is stuff that's not really our fault (e.g. something in the datacenter is broken, or missing network connectivity, etc.pp. - and yes, I've heard of multihoming, but it's not the team's budget and decision to not keep stuff redundant...)

Disclaimer: Ridiculous pager duty was one of the reasons I quit Amazon (combined with massive failures in management, which I'll describe shortly).

The team I was on (retail-related, not AWS), shifted away from the rolling schedule with your counterparts in India taking over for the other 12 hours in order to push the "promote ownership" BS.

The only problem? Management was constantly pushing for new things to get done on extremely tight deadlines (including "emergency features" that needed to be done and in prod in days when they likely required a week of design efforts to get right, never mind actual dev time) so you have two options:

(1) Develop something stable, with good test coverage and the like, and work to fix it if it breaks... and work 20 hours to get it done.

(2) Shit something out as fast as you can and hope it breaks when someone else is on call (who likely will be too busy triaging SEV3-5's during business hours to even think about spending time fixing the root cause of the SEV1/2, rather than mitigating it and moving on...or, even worse (!) (/s) try to get the actual feature owners to fix it) in order to maintain some semblance of work-life balance.

I'll leave it as an exercise to the reader to guess at which route was generally taken.

Do you think the tier 1 thing started after the NYTimes article on them, or before?

I think it started with AWS success

> pager culture where you're awoken in the middle of the night

For multinational corporations, this has never made sense to me. Why do we insist on relying groggy, "at the bottom of their performance curve" engineers to keep your system up, instead of someone in the middle of their work day?

IMO, if you're waking someone up in the middle of the night, something is horribly wrong, and it's not the issue you woke that person up for.

Most don't page people at 1am for the initial page. People only get paged at 1am when the problem has been narrowed down to some specific software and all the people with expertise in that thing are in the same time zone.

Having done Amazon on-call for five years, I can give some background on that.

First, the front-line pager duty is usually hit by an ops eng team first. Each of those teams has an India group and an North America group. If they can't resolve the issue, they page in the developers. Having good Ops Eng guys supporting you is a blessing.

There's also a very strong incentive for teams to write good software and test before deploying when you know damn well that if you're deploying something that doesn't work, you're the guy who will have to deal with it at 3am. Having strong integration and system tests suddenly becomes incredibly important.

Edit: and as dsfyu404ed said, the Ops Eng guys will be smart enough to narrow down where the problem is usually. If you're being paged, it's usually your team's fault.

Amazon on-call engineer here! I'm on-call for three rotations. Two are "Call leader" rotations where a call leader like me is engaged if there's any kind of large event (e.g. Amazon.com orders aren't working, or an AWS service is having issues). I volunteer for those because it's super rewarding to be able to help materially make a tangible difference now and then, wouldn't trade them away for anything. For those rotations, I am paged in the middle of the night, and there's nothing I can really do about that.

The other rotation is for one of my dev-team's software, and I'm hardly ever paged - like once or twice a year max. That's how it's supposed to be: by having skin in the game and being directly on the hook for the reliability of what we produce, I'm sure that we automate to a much greater degree than we might otherwise. We have a culture of strong ownership: it's not uncommon to see senior engineers elect to be engaged on all of their team's live issues, even if they aren't on-call themselves.

Some other teams make different trade-offs, and have follow-the-sun rotations or engagements that are more reliant on humans following run-books than automations. There can be a place for that, but it gets old quickly and is not scalable. Good teams will prioritize the work to get out of that above almost anything.

> The other rotation is for one of my dev-team's software, and I'm hardly ever paged - like once or twice a year max.

I'm in an on-call rotation (not at AMZ) where I hardly ever get paged, and it's cold comfort. I still have to carry my laptop with me everywhere I go, make sure I have access to the Internet, make sure I don't have one too many at the bar on Saturday night, etc.

I've experienced that when the coders are getting paged they'll focus on quality and writing better proactive tests. But when you throw the ops responsilities over to another team then that other team gets to decide how and when new code gets shipped which slows down the whole operation. I've experienced that Dev vs. Ops arrangement too where the ops team was incentivized to maintain stability over allowing new code to be shipped which resulted in no code being shipped! When devs own the ops responsibility too then things move faster.

> AFAIK they also have a brutal pager culture where you're awoken in the middle of the night.

That's a fairly normal part of Operations work and has been for decades. Most places (I assume Amazon is one of them) will have a pool of lower-tier admins on deck 24/7 who handle and investigate 99% of the issues that crop up. But for that 1% they can't deal with, then they need to be able to reach someone who CAN deal with it.

It's much cheaper to occasionally call an SME every once in a while when they're needed, than it is to hire another 3 or 4 of them to cover a 24 hour roster.

I've been called out in the middle of the night to fix someone else's mess more times than I can count, so I'm 100% behind Amazon's cultural reasons for sticking to this sort of on-call system as well.

Thanks for writing this, and with all due respect:

This exact article could have been copy/pasted from any of the other big-tech companies. What makes Amazon different?

The way Amazon does one-pagers is sort of unique.

You mean on six pages? ;)

The 20 minute quiet reading in the beginning is kinda different.

At the other companies I've worked for, you were expected to come to the meeting having read the docs sent over email.

The Amazon way appeals to me because in my experience pretty much no one ever does the pre-reading.

And even if they do chances are they've context switched enough since reading that most of the details are lost.

> At the other companies I've worked for, you were expected to come to the meeting having read the docs sent over email.

It may be expected but large numbers of people don't do it. The 20 min. thing is a bit odd but I can see how it could be a net positive.

Yeah, although the 20 minute reading seems better to me. For one thing, it means everyone in the meeting has actually read the docs. But it also means that it's fresh in everyone's mind when the meeting actually starts.

The author says as much:

> ...but at the end of the day it’s just another high-tech company, nothing surprising.

So, basically:

"If you like being shunt into small teams to work on small projects that take too long and go nowhere, being unable to effect larger changes that actually improve things upon the sprawling mass of wasted effort and reinvented wheels, no long term organization or planning or realistic goals, no semblance of cross-group communication or organization, total enslavement to a disjointed hierarchical mess of disparate initiatives that are ignorant of one another (to say nothing of even caring to help each other), and wasting half your time supporting legacy systems that nobody has the energy or balls to finally put in the dirt, welcome to working at a big tech company."

I might be a little cynical.

I've read many posts here on HN about how terrible working for Amazon is compared to other big tech companies, so this makes for some balance, at least.

> I could go on about diversity ... and so on, but at the end of the day it’s just another high-tech company, nothing surprising.

Dealbreaker. "On balance" in this industry isn't good enough, and if any company can take a leadership role in promoting diversity, it's one of the Big Four.

The site is now offline, probably due to load.

Google cache link for the lazy: https://webcache.googleusercontent.com/search?q=cache:70Vxjs...

Thanks for the mirror, but my God this is a tepid article.

TL;DR - Working at Amazon AWS is just fine and pretty much what you'd expect.

While informative, this post is extremely non-contentious, and i suspect on purpose. Much of what is said here can be said about most companies that are at scale. Probably by design b/c if his employer or manager found out about it, i am sure there would be backlash.

I heard they will actually cut your base pay if stock goes up “too much”. Is this true?

This is not true. However there is a target total compensation range for your [level X review ratings] and if the stock price growth results in your future total compensation projecting above your target compensation range you won't get refresh RSU grants.

What’s the justification though?

Reply on news.ycombinator.com