Amazon built a new unit to fix its engineering culture
businessinsider.com
businessinsider.com
Internal developer experience is extremely important. One codebase I worked on used to have a 24-hour, nondeterministic CI run. A teammate brought that time down to ~45 minutes (it's a big codebase with a ton of tests) and made it deterministic. Everything got better:
* Faster turnaround time for external contributions (it's OSS)
* Ability to be relied on to get a critical fix in and tested in under an hour
* Much faster local build times
* Removed reliance on some old busted internal infrastructure so we could use much better, different infrastructure, and make the build/test runs the same across any OS rather than rely on windows so much
* Everyone was happier in general
If we could have gotten a full CI run down from 45 minutes to 15 minutes, that'd be great. And there was still quite a lot of technical debt in the tests themselves, since they should be able to run a lot faster than they did. But even with that in mind, the difference was night and day.
It’s nice and short but needs one more sentence in the middle. This is up your alley — great! But do you want to compare notes? Do you want to get hired? What’s in it for someone from Amazon to respond (or harder, since you’re asking for an intro: why would someone send this to a friend?)
I get a lot of messages like this and usually ignore them because they lack any reason to respond.
Most large companies have teams for this. I've found that the earlier in a company's life you add someone dedicated to this the better.
* Got buy-in from my coworkers
* Collected data on how much wasted dev time there was
* Collected data on what we would gain (be specific: time to live, faster release cadence, more releases, whatever is real and appropriate)
* Collected data on what clients would gain (errors more likely to be caught before release, more features, etc)
* Present to boss, refine message, keep going up the ladder
MVP for a PoC, in my opinion:
* Come up with project goals as a team
* Sane branching strategy. Can be done at the team level if needed by making a team-level "main"
* Standalone (meaning no-deploy, "devops" framework independent) unit testing system. Use the one that comes with your language if it doesn't stink. Add unit tests to new code.
* Basic build & test system. TeamCity is easy to deploy, configure, and maintain on-prem, Jenkins supports complex use cases. Automatically run on commit to the team-level main. Allow devs to run on-demand for their feature branches. Use other systems (GitHub Actions, Azure DevOps, etc) if you have the budget.
Keep it simple. Don't worry about pull requests and approval chains at this point. Automatically build it, test it, and put it someplace it could be deployed.
Reason being everyone I know who got in around 2018 has options that cost a fraction of the current stock price. The job is still boring/miserable but when you are clearing 600k+ annual income...
Now if you join, you can bet its a far worse deal and you will make far less than those who just happened to join earlier, even if they're checked out...
There in lies one of their problems, the incentives are for checked out people to stay and 'click buttons' while performing as much CYA as possible.
The other one is that compensation and competition is sobering up. I have one former colleague who claims that they earn 3x as much working for a US company than they did our previous job, but I don't see much exorbitant wages being thrown around. Second, that company he works for now has set a hiring stop; the explosive growth and demand from US based tech companies is slowing down or coming to a halt.
> In an email to Insider, Amazon's spokesperson confirmed the existence of the team, saying it's part of an effort to maintain its Day 1 culture.
This terminology is bullshit. As someone who was actually there at Day 1 of amzn, it's completely clear that the term "Day 1 culture" is truly meaningless. It refers not to a point in time, but to some management-imposed aspiration in which things never actually settle down to a routine, but instead everyone feels "empowered" by running around constantly reinventing everything. Of course, this is sold as "you can still make a big difference here", when in reality, you almost certainly cannot.
Amazon's actual Day 1 culture meant bringing up a couple of SPARCstations and getting the T1 line functioning.
It's annoying to devs but should be alarming to management. You're literally spending 2x to 3x the required amount on your labor because of your inefficiencies. Personally I found it easy to convince management of changes like this when you pitch it like: "How would you like to reduce your labor costs by 2x". Now granted many people at these places are salaried which makes this a bit of a moot point but most software shops contract out alot of work, and that's where you're really paying for your inefficiencies.
I don’t work at Amazon, but I’m sure there’s plenty for folks to do.
Suggesting efficiencies that would reduce headcount is a good way to get on executive leadership's threat radar.
Man, I do not miss working for a fed contractor.
Anytime somebody say's something like "does it really need to be this hard" or "all you've got to do..." should make sure they fully understand the rationale behind the requirements. A lot of times, those requirements can be met in a more efficient manner, but sometimes the seemingly "efficient" path will cause more second-order problems down the road because of those blind spots. What you often find in both camps (those bluntly instituting the requirements and those saying they should go away) is a lack of understanding of the entire system dynamics.
Many times that process step is just arbitrarily layered onto an existing process. If you do that enough times we get the inefficiencies that we tend to think of when we say "bureaucracy." That creates a risk in itself, namely a productivity risk. But that stems from my earlier point: when process steps are just layered on without thinking about the second-order effects, it creates a bad process by incurring outsized risk. But the converse is also true: removing process steps without understanding the rationale behind them can also increase risks.
You seem to equate bureaucratic steps with needlessness. I'm not sure I agree. I tend to think they are there for a reason, but possibly just implemented poorly. Same goes for decisions to remove steps from a process.
I remember working for an insurance company in IT and we would always get a type of ticket that we were told to re-assign to another team. When I asked the obvious question of "why doesn't that team we send it to get it in the first place", I got all sorts of answers basically culminating in: "I don't know but it's the process and it's important". I eventually got someone to walk me through it and admit through their own answers that there was really no good reason to keep this process manual, as it must have been a holdover from when they had an older ticketing system. Even with all this it was impossible to change that process (at least while I was there) because the answer was always: "yeah your idea sounds good, I don't really see where we could break anything....but what if we do break something, let's just keep it as is".
“If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”
This means somebody needs to first understand the reasoning behind it. If, as in your example, that rationale no longer holds, getting rid of that step is perfectly inline with the principle. I would argue that if somebody truly understood the process there would be no issue with changing it.
This shouldn’t make a difference because in theory they would be doing 2x more work which is essentially cutting their salary in half. I realize that the dev team being twice as efficient doesn’t necessarily double the unit’s revenue but it’s some multiple.
I'm starting to feel very old
Is Java considered legacy now?
Java hits a sweet spot of performance, tooling, and productivity that few other ecosystems can. It's biggest competitors are probably C# and golang, and, for various reasons, it is holding its own just fine against them.
That’s legacy Java.
I largely agree that for me, working here feels "day 2", which just means it feels the same as working for any other mid to large sized corporation. Lots of meetings, building consensus, tasks that amount to overhead, and adjusting slowly moving pieces to advance whatever the corporate agenda is. Many of these things are directly contradictory to amazon's stated leadership principles (core values) but work happens at the scope of teams not at the scope of leadership principles.
Some of it is team-dependent (not unrelated, my team has really good work life balance. I slog through annoying corporate BS 9-5, but don't work much beyond that) and some of it is related to the builder tools as mentioned in the article.
I mean, literally anybody anywhere can do this. Whether you get fired depends on your performance evaluations. Like any organization, the biggest reasons to stay late is (a) too high expectations were set, (b) failures to re-adjust to new information (e.g. we had to pull you off your project for 2 weeks but still expect you to make your deadline), (c) paging / oncall. At AWS, I'd say (c) is the biggest risk, but (a) and (b) can also be present at any team of course.
> Is it just the luck of the draw who ends up on the teams with good managers
Yeah, that's my impression. When you're already in Amazon though, you're empowered because you have a much greater ability to gather information about a team before deciding if you want to transfer. For example, message/meet with devs on the team, look at their ticket queue to see how often the oncall gets paged, and so on.
> Or can you easily switch teams until you find a good one?
I don't know about "until you find a good one", but transferring is pretty normalized. If you've been here a year or two, it's normal to want to transfer, and even before that it's not impossible. You can't transfer if you're on a performance improvement plan, but otherwise it's pretty straightforward
In a way I kind of respect Amazon for treating their white collar tech staff as poorly as their blue collar warehouse staff.
This isn't always the case. Ask a TAM. With the right blend of customers I've known some to work about 10 hours a week.
May be because their performance review prioritizes employee contributions that impact bottom line immediately instead of building infra that might improve developer productivity?
(I don’t know that this is the case with Amazon: I’m just providing an anecdote based on my observations at places I’ve worked.)
There was a clear culture shock for some of the more seasoned Amazonians (and you could see a fall out effect with some "mails to your manager" every now and then if you crossed a Grand Poobah the wrong way), but to all of us "newer" hires, the times of adversarial review were just a story.
Anecdotally, I can say with confidence that some teams have a lot of autonomy in deciding to follow (or not follow) company-wide practices / guidance like what you've posted (obviously, only a few do it, since non-compliance isn't officially blessed). I was lucky enough to work in a team which was much better than baseline. But I try not to let that colour my memory - I heard and saw plenty of horror stories, and I saw plenty of people (including in my leadership) trying to do the right thing. Like most things, there are shades of grey here.
Andy, you've known about the culture problems for years, the fact that you're just now acknowledging them is only because the business is hurting, and burning out all the talent. You're scared shareholders will get involved if this isn't fixed. Sadly, this won't fix anything, this is the culture YOU built.
* As long as it doesn't interfere with the share price and exec bonuses
I would argue you can't fix the tech employment without fixing the rest. One bleeds into the other. I would never consider working for a company that can't treat it's more replaceable employees well.
Stripe, as it has scaled, has ran into challenges and invests in the same efforts. FB internal tooling was (is?) a mess. It goes on. The surprise here is it's such a late investment from Amazon, but it's worked for them thus far. I'm guessing the returns on +1 engineer has plateaued or declined, leading to this effort.
At the same time, a friend was interviewing for the exact same senior engineering role and received an offer. The total compensation he was offered in his first year was nearly double my pay. He didn’t even take it - another company offered him more.
Is petitioning a process in AWS? What does boomerang mean here?
Petitioning isn't a process in AWS, but you can ask your manager for a raise if your projected total compensation isn't to your satisfaction.
Boomerang means you leave the company, get another job, and then come back to the company with much higher compensation. It's a way for otherwise happy employees to fight salary compression.
I’ve also seen a scenario where someone has trouble getting promoted and leaves to boomerang back at a higher level.
Of course the company hates to give existing or recently departed employees raises of any type so they closed that loophole.
Fair or not, this is currently the optimal career/comp progression strategy, although recent hiring downturns might change the meta(heh)game to favor internal candidates. We shall see.
You guys get to click buttons??
https://aws.amazon.com/builders-library/ among many other AWS docs that use the builders language.
I may like some of the services like s3 but will never rely on AWS for anything more that that. Logging into the aws console makes my blood pressure go up.
For instance : if build times are really bad, maybe it's because developers don't feel empowered to fix this as part of their day-to-day work.
The need for these DevX groups comes from the fact that at Amazon's scale, there's lots of powerful tools, such as a massive internal build system and such.
So, one answer is that you need DevX groups, because it makes no sense to have each of the thousands of developers learn how to tinker/fix the huge custom amazon build system (IE empowering individual devs).
A second answer might be that there shouldn't be massive internal systems in the first place.
A lot of the brilliance of AWS was reducing the differentiation between internal and external teams as consumers of services[0]. The buildup of massive internal systems probably shows that Amazon hasn't gone far enough, and DevX failures are just a symptom.
[0] related https://www.alexanderjarvis.com/steve-yegges-famous-rant-on-...
OTOH if your build system needs a lot of tinkering/deep knowledge to be used effectively, maybe it needs a redesign. This would be a job for the tools team rather than DevX.
Culture fit matters and you can't fix the culture when thats what they are hiring for.
Having self contained workspaces with all dependencies in any language, fine grained ability to replace any package with a local build, and the simplicity of package config (for most languages) was awesome.
Having to use _n_ language dependent build systems without interface pinning or reproducible builds since leaving sucked, imho. Checking out a package and running `bb` to get an artifact was just so satisfying.
Having to set up a million dependencies just to begin to start working on a codebase is a nightmare, and Brazil makes that easy.
My own opinion is that the Brazil had far too many running parts. I didn't like that reproducibility required the tool interacting with 3-4 different services, and I'm much more a fan of having the source code repository act as a single source of truth for reproducibility. Plus, using Brazil meant you had to use their entire tech stack, where GitLab/GitHub beat them by far.
But, FWIW, I actually like Brazil.
You mean the country, not the tool, right?
I eventually boomeranged back to Amazon and have found the middle ground. When used right, Brazil solves a lot of problems and day-to-day I enjoy working with it much more than not. The trouble is that almost all teams don't know how to use it and tie themselves up in knots of their own making. Third party software is also a problem. I hope Peru comes through, but so far I haven't worked with it, for good or for bad.
Their reputation is catching up with them and their lack of stock growth is highlighting their comp deficiencies.
I left Amazon due to it being a total shit show (for me and my mental well-being but also for the team I was on). Went to Microsoft and found it to be absolutely wonderful. I was working at 6pm one day and my manager, as he was leaving, chided me and said, "Go home to your family. Work will be here in the morning."
I have had friends/colleagues leave to go to Amazon for the TC. One even tried to get me to go over there for a PE role that likely would have meant a huge pay increase. But I value my sanity and have no plans to go back there ever.
You had (have) a good manager. A manager who actively cares about the work life balance of their team is more valuable than a huge chunk of TC.
It comes with the age of the company. Unlike start-up land, loads of people in Microsoft have families and kids, and value their family more than they value work. It's normal to leave work on time, and it's concerning if someone stays late.
"its incredibly high pressure and demanding, but worth it"
The money is the drive. And I worry about going for a money job that I hate because getting used to the lifestyle will mean it is that much harder to leave. I am already in that boat where leaving my current job basically means a pay cut or going to a FAANG. It is very limiting especially with family pressure.
- Hacky code with quick turn around times.
- Lots of reporting and CYA.
- Very conservative behavior (nobody's going to go out on a limb to write something new because they're too busy churning through those "high pressure" Jira tickets).
In the long run you just have an army of engineers, their backs whipped raw, toiling frantically at a mountain of tech debt.
And yes, I've met two people who worked at AWS and liked it, essentially because they were hyper-specialized and had the rare cushy gig there. Everyone else loathed it within months, and no one lasted longer than a few years because AWS squeezes out 'productivity' with everything short of rusty pliers.
Most devs I know see their endless recruiting emails as a trope at this point
They've repeatedly stolen the IP of top selling product companies, then you've got this stuff https://www.forbes.com/sites/davidphelan/2019/04/12/amazon-c...
Not to mention all the heinous shit (unconstitutional wars, dragnet spying) they help our military and intelligence services do https://aws.amazon.com/government-education/defense/
Their abuse of their workers from warehouse to engineering is just one piece. As I tell their recruiters I'd rather eat a cactus than work for them.
wow, hardcore.
lol. whats their response?
That's not much of a statement https://en.wikipedia.org/wiki/Nopal
We have those rules because otherwise HN would basically all flamewar all the time. And not even fresh flamewars.
I'm not defending Amazon, btw. Their behavior can be inflammatory and your comment can also be inflammatory. That in fact is how flamewars work.
A discussion about their engineering culture without discussing the larger related problems is myopic. It would be like discussing problems with the logistics of moving people to the gulags and you’re saying we shouldn’t get off the topic of transportation.
There shouldn’t be any flame “war” because everything I said is correct. I don’t think there’s any popular defense of these behaviors from Amazon, flame wars are characterized by popular opposing schools of thought.
Moreover, you listed them using the denunciatory political rhetoric that is most antithetical to the curious conversation we want here. No doubt BigCo deserves it, or at least you feel they do (and I'm sure you have many good points), but we're trying to optimize for one specific thing here (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...).
It sounds like you define flamewar more narrowly, which is fine—I'm talking about the kind of high-indignation-low-information internet discussion that evokes angry-reflexive responses from people instead of curious-reflective ones. Internet threads have a strong tendency to end up in that state, and it's exactly the opposite of what we're going for here (see that previous link), so we end up expending a lot of energy to try to stave it off.
People often mention "correctness" (or "truth" or "facts") to justify breaking the site guidelines, but of course there are plenty of ways to be aggressive, flamebaity, or just plain off topic* while nevertheless saying correct things. Correctness is great, but those other qualities are undesirable regardless of how correct you are or feel you are. Here are a couple recent explanations on this last point if you or anyone want more:
https://news.ycombinator.com/item?id=32909407
https://news.ycombinator.com/item?id=32697044
Lots more here: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
* Not all offtopicness is bad; whimsical offtopicness is usually fine. It's the generic tangents that are bad, because they drag down discussion quality.