Hacker News Reader

Comment by salil999 | Hacker News Reader

Former AWS engineer here.

I worked on a pretty critical product in AWS (big AWS service with lots of traffic) and I can safely say that it's totally up to your manager and pre-existing conditions which make up the job. My manager was great as a person but would always lack in my career-oriented goals (bigger projects, promotions, etc)

But what really sucked for me was the pre-existing conditions. Our on-call was pretty bad (40-60 tickets a week) and there was very little investment being put in to improve it. We had a lot of little scripts here and there which would solve extremely specific situations but no focus was ever put on in building a general framework or trying to reduce the ticket count. This often led to engineers taking the day off after their on-call due to the load and honestly it made people quite grumpy. And upper management was always much more interested in feature delivery since the focus was always on promotions and the more you delivered the better it looked for your manager. So now you have engineers with such a terrible on-call load along with pressure to deliver new features and projects within the atrocious tight deadlines that would be set. It was, to be blunt, a shit show.

Code quality was atrocious. We had one enormous Java method (>1000 lines) which would take care of nearly every single request coming into our service... With only about 7-8 unit tests. It was so difficult to get even basic things done to the point where any ticket that needed to be done would take a minimum of 4-5 days regardless of complexity. And of course managers and senior engineers would estimate small tickets to take around 1-2 days and then be shocked when 2 days later it's not even close to being finished. I will give Amazon credit that they do grill design reviews pretty harshly so those are done well in general. But code reviewers didn't care about quality or best practice. If it works then ship it.

I'm just not 100% sure about the whole PIP scene. Our service was extremely critical and we were extremely understaffed. So I don't think it applied to anyone in our org but I know of other teams who would have no issues in taking in a fresh college grad, making them do work for 6-12 months and then just randomly putting them on PIP. Sad but I've seen it happen a few times in my time there.

I'm glad I got the Amazon stamp on my resume and left. When I left, more than half my team and my manager quit around the same time too. It was definitely a wild experience.

View on HN

Current AWS engineer here, can confirm. I'm absolutely broken. I'd second the point about the manager and pre-existing conditions making up the job. It's not clear to me if it's endemic, these are big orgs with teams run very differently.

That being said, the on-call sucks. It's really awful, and something I've never seen before. It's also typically the primary cause of team churn for people in my org. This varies, as I've seen other teams stacked with L6 engineers with very little churn (4-8 years of tenure each). This is very much a pit of despair of our own making, but I still haven't figured out how teams like mine get out of it. My own view is that normalization of deviance means that engineers who've only worked at AWS just accept that getting paged many times in a week at awful hours for false positives is OK.

There's certainly a view that the only way to get promoted (which is incredibly difficult) is to create new features or products. You read this of many orgs though, not just AWS. I'm not convinced it's fair to single out AWS. It can be endemic in some teams, and I've certainly worked with engineers who are clearly only focused on shining for promotion.

The worst I've seen was a team go from 8 engineers with > 2+ yrs tenure down to 4 people with < 6 months, over the course of a couple of months. This was for an enormous product. That team had a very tough 4th quarter.

AWS does handle operations failure incredibly well. If you've hopped on a LSE call before, the execution to identify, mitigate and review correction of error (COE) is world class. Doc / design review is also very thorough.

There's ample opportunity to learn a lot during your time at AWS, and many engineers have carved out incredible careers in this place. Just go in with your eyes wide open.

What is LSE?

Quote-Googling produced "Large Scale Event"

"large scale event" or something to that affect - these issues are visible to everyone company-wide

Large scale event, something that has big impact on services.

London School of Economics ;)

I’m sorry you’re having a rough time. Hang in there, and if you move on, good luck!

Thank you, I appreciate you saying this. I don't want to give the impression it's all awful though. We are reasonably well compensated (you do a lot better if you're based in certain countries than others).

There are part of this job that were a lot harder than I could have imagined. The aim of my earlier comment was to make this clear to people considering AWS. When the hiring manager interviews you and mentions there's "an on-call component to this job", realise that it _can_ be severe. The on-call time is also unpaid; it's also very difficult to spend significant time on improving this situation, if that's possible at all. Other comments have done a better job of describing this.

There are parts of the job that are fantastic. You have access to some outstanding engineers (this is also the case at many other companies though). Almost all of the principal engineers I've interacted with have been very generous with their time and knowledge. I thoroughly enjoy the Principals of Amazons talks, and subsequent discussions. I've also had the opportunity to be able to look very deeply at technical problems (this is a direct result of my manager). Having worked at a number of SMEs before, this wouldn't have been the case. You also work on systems being used by so many people (this is mostly wonderful in hindsight), which having worked on products that have evaporated into the ether in the past, is rewarding.

It's not for everyone, and it's certainly not forever at AWS. My guess is I'll walk away with some scars and a much better idea of what I want I don't want to spend my time doing.

You'll learn a lot and cry a lot.

> The on-call time is also unpaid

This is why on-call should never be unpaid. There is no incentive to reduce cost and improve that way.

Sorry to hear about your awful job, is this only in engineering or does it affect professional services like solutions architects etc too?

ProServe and SAs have little-to-no on-call responsibilities, but the general workload issues and mindsets affect those teams as well. Just as an example, resource management (aka staffing) and project scoping are things that AWS Sales is absolutely fucking awful at, and are things in particular that other consulting companies have figured out decades ago, but AWS does nothing to improve the shitty staffing processes because they have essentially just thrown up their hands and think the inefficient, inaccurate, and incredibly stress-inducing process is normal.

Of course, I guess there's no perfect isolation. A friend at Amazon set me up a video call last year with an SA at AWS (Seattle/NW) so I could inquire about working there. It sounded pretty cool but I wasn't sure. I didn't think that AWS consulting was much of a thing, since my experience client-side has been that AWS will advise but not build. Anyway, irrelevant to this thread. Not sure consulting is much better away from AWS as in my world, it's 90% dependent on the client you're with and boy have I had some awful projects.

Based on my experience working on different teams and interfacing with the SA teams (but not actually working as an SA, mind you), being an SA at AWS seems to be one of the better positions in terms of workload, limited on-call responsibilities, etc, but you do need to be very comfortable with constant customer-facing responsibilities (which can sometimes be better than on-call, and sometimes can be worse than on-call) and the team is still dragged down by the general Amazon/AWS culture.

> pit of despair of our own making, but I still haven't figured out how teams like mine get out of it.

I’ve only seen two ways out: 1. Team implodes when everybody leaves, reorg follows making it some other team’s problem 2. Management recognizes it’s a problem, takes it seriously by staffing the team with enough people to sustainably address the tech debt/operational load AND build new features

Current Amazon engineer: it's far and away the most incompetently run bureaucracy with self-defeating dysfunctions forced on the huge number of layers. The recent "leaks" to Business Insider, pointed out to me by coworkers, are exactly what we see.

Everything the parent comment mentioned is exactly what we see. Laughably, our middle manager berates us as incapable noobs, entirely unaware that some of us at least actually have been very valued and trusted and dependable hard core results driven engineers for decades. It's like a failed brainwashing attempt, and it's embarrassing to be associated with such idiocy.

I contrast it with excellent experiences at Sun, Motorola, Apple and others over the past 20+ years, where in some cases I had very high engineering ranks with fabulous results, in very well run and just healthy orgs.

I do believe there's intentional treatment of engineers as fungible assets, because the engineering quality is so shoddy and the business plan prioritizes maintaining system uptime, with no true priority given to tech debt removal, which actually would in short order surface measurable benefits.

> I contrast it with excellent experiences at Sun, Motorola, Apple and others over the past 20+ years, where in some cases I had very high engineering ranks with fabulous results, in very well run and just healthy orgs.

I can understand how junior devs end up in these positions for a while and stay on the ride, but what is making someone with your level of experience you stick to Amazon? Is the compensation that good?

Compensation seems good, but I'm there because I got fooled and plan to quit very soon, when graciously (not screw team-mates) feasible, because it's just a waste of time that could be spent helping positive efforts in healthy orgs.

I have felt that way before, but it’s better just to quit ASAP. It’s the company that’s making your teammates suffer and their also better off just quitting.

I live by the motto "I work for money and appreciation, in that order. If you want loyalty, buy a dog!" It has served me well and removed a LOT of stress. I walked out of one crap-hole with a yelling management style on on week's notice. They told me it was unprofessional, and I laughed in the VP's face and told him he was lucky he got a week.

That is also my motto at work. You know what's unprofessional? Managers screaming at people. The fact that they thought dysfunction like that is OK but one week's notice is "unprofessional" tells you everything you need to know.

> ...I'm there because I got fooled and plan to quit very soon, when graciously (not screw team-mates) feasible, because it's just a waste of time that could be spent helping positive efforts in healthy orgs.

Quit now, and encourage your teammates to do likewise. I understand where you're coming from, but the attitude you have can be a source of exploitation (e.g. using loyalty to peers to keep you while the org screws all of you more).

There is usually on hire cash, that you must return if you don't work for a full year to prevent smart moves like this.

"Unique technical challenges"

Some of the problems you can work on in such an environment are interesting.

Combine that with the fact that you don't see this before joining and that some.manager s might be able to shield some of it you are where you are.

And then throwing hands in the air and leaving is a step, as most other companies have "smaller" problems to solve and "also bad management" ...

Only Apple is on Amazon's tier of technical success, and half of Apple's tech success is from non software.

So on what basis ia Amazon incompetent at software? It sounds like Amazon is results driven, and pretty well-unit-tested code isn't critical for that.

Sure, why test code when you can throw expendable people at it instead?

OP made no mention of software and mentioned Sun and Motorola, so I don't think they made any point specifically about software.

It may also be relevant to point out that Amazon is first and foremost a retailer, unlike the other companies mentioned.

> Amazon is first and foremost a retailer

I feel these days that it's more correct to say Amazon was first and foremost a retailer, but these days they are just UberMegaCorp across so many businesses (Amazon site proper, AWS, WholeFoods, Kindle, Echo/Alexa, movie studios, etc. etc.) that it's hard to say they are foremost a retailer.

I have a similar question, though, in that I've heard lots of nightmare anecdotes about Amazon code quality, but obviously whatever they are doing is working on some level.

There a plenty of things that decouple incompetency in their software org from business success:

1) Amazon has been incredibly successful frontrunning two MASSIVE verticals: online retail, and cloud computing. They were early winners and invested an INSANE amount of money to be #1. That gives them a lot of leeway to fuck up and still be on top. Even if they stopped doing anything at all but maintaining whatever shit show is going on behind the scenes, it'll be a while before they aren't #1 in at least one of those two huge markets.

2) Tech debt doesn't hurt you today, it hurts you tomorrow. You can cut corners and get products out the door fast, but eventually features and products that should take X amount of time, instead take 5X. Given their lead in their units heavily reliant on software, I doubt it would hurt them anytime soon.

3) You can shit on a lot of people for a long time before it hurts hiring when your stock has shot up the way Amazon's has. Money will make people do dumb things.

4) It takes a long time for poor hiring and management practices to incapacitate an organization the size of Amazon. Turn over at the bottom may be high, but I doubt it's the same at the top, at least up until the past few years. When leadership starts turning over more you'll see a lot more business mistakes.

well the good news (for amazon) is that I used to work at a place where I was trying to build a cloud. And management suffered from EXACTLY THE SAME PROBLEMS (though for different structural reasons, there was no "build a feature" incentive). And then we hired someone who used to work on amazon cloud who came with rave reviews, even though he completely failed my interview, which basically tested "do you read the spec?". Then I told him to take notes while I was onboarding him, which he didn't do, and then I quit, because of the management problems.

> the engineering quality is so shoddy

What stuns me is just how well AWS works. I mean, did you ever try to use Azure in 2015? Hell, even GCP was worse in that era. Now, it's on-par, but IMO "more confusing" which beggars belief considering how confusing AWS is.

Yeah it’s surprising that despite the negative comments in this thread, AWS works almost flawlessly for stuff that matters (uptime, reliability etc.). Of course it’s painful to work with sometimes, but so are other cloud vendors. There’s literally zero incentive to change things internally until external KPIs start turning red.

IMHO that's the down side of Amazon's customer obsession. As long as it works for the customer, nothing will change. Regardless of internal benefits. And that system quite obviously works.

Disclaimer: Former logistics Amazonian.

I feel for all of you in this thread that work at Amazon and complain of the poor conditions. There are no shortage of stories that point to a very large problem in that company, for certain. However, I believe the underlying theme, if there is one, is incompetent Management and Leadership.

This is systemic in Technology companies because, to me, the “bean counters” try to run the companies like a basic Manufacturing organization of commodities where line items and people are substitutable things and can be dialed and tweaked at the planning stage.

You nailed it. At the foundation of the dysfunction at most technology companies is the inability of the leaders to understand that software heavy products just can’t be estimated accurately by traditional “bean counting” methods.

Not just 2015. A lot of people, including me, would love for azure to be on par. Unfortunately it isn’t. Not in reliability, features, or even documentation. Hopefully it’ll get there.

GCP is an odd duck. It really wants you to do things its way. But if you are doing a multi-cloud strategy the last thing you want is to go all in on the GCP way. Also they just don’t seem to treat customers as seriously as Amazon and Azure. It isn’t life or death for Google and it shows.

Oh yeah, gcp customer service was atrocious. Took several months to find out that we couldn't do something undocumented but not obviously impossible.

> Everything the parent comment mentioned is exactly what we see.

Current sde1 bluebadge. I also see exactly what the parent comment mentioned. I also saw a few nasty episodes such as L6 managers quitting in less than a year in because they hated the pressure they were subjected to, and a yellowbadge say during a all-hands to a bunch of newly arrivals that he was at the company when they joined and he will still be at the company when they all left. I know of a team where SDE2s and SDE3s felt the need to pull in all-nighters and even work the entire weekend to meet the deadlines that their L7 leadership pulled out of their ass. I myself felt compelled to work close to 12 hours a day and weekends during a period just because my senior manager wanted to shine. And yes, I've already see a fair share of colleagues disappear.

Of course YMMV but I'm baffled why I'm seeing accurate descriptions of what I'm personally experiencing on a daily basis being faced with such a sense of incredulity.

> because the engineering quality is so shoddy and the business plan prioritizes maintaining system uptime,

I've created this throwaway account just to comment on the issue of engineering quality being shoddy.

It's true that engineering quality is shoddy but the problem is caused by Amazon's structure and not by the engineers themselves. For example, each and every single engineer is expected to deliver major tasks alone as part of their evaluation process, and this means that by design we are placed on tasks where we are way over our head and we are still expected to deliver things out of thin air. This sort of trial by fire is sold under the premise of allowing us to deliver consistently at the next level before being considered for a promotion, but in practice if you screw up that's expected to also be held against you.

Former AWS/Alexa engineer. Totally forgot about the whole badge color thing. Can't stand that crap.

Things are very dependent on team and manager and org but I will say that one of the things I'll never miss is our Re:Invent launch. That was horrible. Our team (startup acquisition) basically had a giant retro after the launch to figure out how we could avoid doing that again. Most of the startup members left the team of the company 2 years in. I learned a lot.

What do the badge colors mean?

Length of tenure

Former contractor here...

blue = full-time/permanent employee

white = seasonal/temp employee

yellow = vendors (and DSP employees, if I remember right).

It's more or less their way of tagging you with a yellow star so they know who they can treat like shit. Hah, just kidding, they treat everyone like shit.

This is incorrect: https://blog.aboutamazon.eu/discover-whats-behind-the-amazon....

I'm genuinely not sure what those are. They appear to have been proposed at some Seattle facility: https://www.seattletimes.com/business/amazon/amazons-sleek-n...

I've worked at two different facilities in CA and never once seen those. We still used the white/yellow/blue badge scheme.

Are there any ambassadors in this thread who know what those are?

If you saw any blue badge employees with a yellow/red/purple border around their badge, then that signifies how long they've been at the company.

Saying someone is a yellow badge badge is saying that they are a full time employee with 5-9 years of tenure, most SDEs don't work with contractors so we don't talk about yellow badge contractors all that often.

If you're interested in this sort of thing, the color is actually supposed to be orange, but the color came out yellow.

Honestly, I never ran into many blue badges down in logistics. Mostly all white and yellow badges, with a few exceptions... and most of the time I was in and out so fast I wouldn't have noticed any colored border on their badge. The only time I would've interacted with a blue badge is if there was a problem on-route.

TL;DR DSP drivers don't encounter many blue badges.

The article you linked is from five years ago when the new badges were being initially tested. They have been rolled out for years with color photos. Both styles still work and if you don't go into one of a few big offices you might not have easy access to the newer style.

The yellow stars shouldn't be taken lightly. Don't compare badge colors with what Nazis used for holocaust.

Not speculating as to the reason why mtmail is being downvoted, but for those wondering, the reason the nazi Yellow Badge comparison is almost always off-limits is because there's really only one hard and fast rule as to whether the comparison is valid:

Is an entity imposing an identification system for quickly singling out populations with invisible but immutable characteristics? If so, the Yellow Badge comparison is apt because that's exactly what the nazis did with their badging system.

Amazon identifying people within certain buckets of employment doesn't fit this comparison because opportunities exist (in theory) for people to take other roles, to quit, or to be terminated, making Amazon badges identifiers of a mutable characteristic.

Now, is badging people by employment class a crappy thing to do? Probably. Easy way to enable eyeballing a worker and denying certain perks, for instance. But it's also useful in determining whether a person is qualified to hear certain company-confidential information.

---

tl;dr: There's never a good reason to compel visible identification for an immutable trait, but that's also not what Amazon's badging scheme is, so using the comparison risks cheapening the original sin.

Hope this helps anyone reading. And to any historians who probably know better than I do, kindly fact-check me. I only know enough to be dangerous, not useful.

> I do believe there's intentional treatment of engineers as fungible assets, because the engineering quality is so shoddy and the business plan prioritizes maintaining system uptime, with no true priority given to tech debt removal, which actually would in short order surface measurable benefits.

I mean this how Amazon treats all of its employees. It can't quite lock devs and ops people to a desk and have them piss in bottles, but they would if they could.

Amazon begged to interview, so i did. My friend worked there, and thought I go call her and find out how things worked there. That is when I discovered this forced termination of 30%. I thought it was 15% , but who's counting. So I gave amazon a choice. 2 year contract forced pay contract, if they fired me for any reason, other than fraud, I would collect the full amount. I done this twice.

It was a defunct mobile company. I had the contracting company put it in the paperwork, to my surprise they signed. cause they have multiple times let go of contractors to make C level "bonus". It was known for it. I got some push back, but they signed it, they needed an engineer who understood mobile routing records.

3 months into the contract, they apparently forgot, and let me and everyone go. Reason 'services no longer needed' aka CEO wanted to make bonus. I said ok, then called the contractor's legal dept. After 3 days, i got a call back. Told him look at page 4 line 30.. he read it. Long silence. 5 minutes of silence. "I never seen a clause like this. I will call you back with an answer." Next day, Both lawyers call, one asked would i be interested in returning. I said "fufill the contract terms". "Your check will be delivered in two weeks." Contracting firm, was getting paid too. they were happy and mad. Happy for $$$, mad that they probably lost a semi big client.

I got my check, had a new contract before the 2 weeks.

Fast forword to Amazon. They read it and said no. I declined. They offered little more, but I said I will not work for Amazon until they fix their corporate envirnoment. My friend worked there, she lasted 4 months more, after moving to Seattle, and finding out the hard way that amazon is pure corporate monster.

> So I gave amazon a choice. 2 year contract forced pay contract, if they fired me for any reason, other than fraud, I would collect the full amount. I done this twice.

As someone who has hired contractors and full time employees many times, this is just insane. I don't care if you are Linus Torvalds himself, as an employer I would never sign a contract that is that lopsided, and to be honest I can't believe that any competent company would either. And given the description you have given of the company that was dumb enough to sign this, they definitely don't sound very competent.

So, I mean I guess congratulations that you got 2 companies to sign this, but I can't think of any decently run company that would ever sign something like that.

I think in the end it's all supply v/s demand. If the company needs you really desperately, they'd bend ANY of their policies to get you.

This was pretty common before the financial crisis, though the typical contract length was a year back then. The idea is that it was a 2 way street- the consultant is committing to your company as much as you are committing to them. And if you are a mega corp, honestly one person's salary for a year isn't really that big of a deal. The cost of hiring and firing a FTE is not much less.

Though the length of it is pretty head scratching. The whole point of using consultants was that you needed a short term amount of specific work done, unless you are talking about engaging large consulting companies to deliver a project.

>Our on-call was pretty bad (40-60 tickets a week) and there was very little investment being put in to improve it. We had a lot of little scripts here and there which would solve extremely specific situations but no focus was ever put on in building a general framework or trying to reduce the ticket count.

AWS engineer here and I confirm everything you say, but this quote really struck home with me.

The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues.

I've worked across multiple teams and there is an "institutional ego" at Amazon where everyone, especially L7s+, think that Amazon is the best/smartest company in the world and have an attitude of "if Amazon, the best company ever, hasn't already figured out a way to solve this problem, then it must be an unsolvable problem and we won't even try". The thing is, a lot of these problems are in no way unique to Amazon and many other companies across the world have already found fantastic solutions to reduce things like on-call load. But adopting those solutions would require admitting that other companies were able to solve something that Amazon hasn't, which would hurt the ego.

This all applies to the very issue being talked about in the OP, too. Even managers will vent to you about how their team goes through 50% attrition every year, and how everyone is overloaded and finding new engineers is hard. They just accept 50% attrition as "something that just happens every year" as if having such a shitty team is normal, and there is no movement at all to fix it.

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues.

In my case, direct management seems interested in these issues and understand there are problems we need to fix, but ultimately the feature/product launches always make it into the sprint and the larger bug fixes never do. It's very much "actions speak louder than words".

Related to part of what you said. I've worked in a half dozen different industries; I've worked in little companies with a half-dozen employees and globe spanning companies with tens of thousands and employees. They all think that they have special unique problems that nobody else has - 90% of it is the same problem I've seen in other companies in different industries, with different industry specific acronyms and words.

Every damn job, the same basic problems over and over, with the insistence that these problems are specific to the industry and usually to that specific company and that specific product.

Same is true in academic research (though old-timers catch it often). The common pattern is:

* approach “A” was invented in 1970 or so and didn’t work

* “B” extends “A” in multiple ways and now works

* noobs assume “B” invented “A” and treat “B” as the root of modern knowledge. “B” often has more market presence so noobs (without deep understanding) don’t see the relationship to prior attempts.

Examples:

* AlexNet/deep learning/ML in general

* MapReduce/databases/functional programming primitives

* Docker/chroot “Jail” Containers

* Bitcoin/90s coins/blockchains

Ego and ignorance are rarely a great combination :)

The Emperor's new clothes (TENC). Most people don't do history, especially the Dunning-Kruger afflicted.

Docker (Linux containers) is, like jails: awful, leaky isolation pretending to be virtualization. If you want real resource and security containment, use virtualization. Docker is insecure in so many ways; it's like using PHP to write a TLS library.

"NIH" duplication of effort with "IH" bugs.

SMH. Will they ever learn?

Can you share your knowledge or reading materials for how to reduce on-call load?

I’ve worked at a number of big companies but all the problems driving the oncall load seemed, at best, domain specific if not application specific with highly variable fix times and unpredictable occurrence (eg started becoming more of a problem due to unrelated change X). As a result each team has to decide the cost of fixing the pain vs focusing on other things.

If there’s actually best-practices here that help that we’re not already doing, I’d be extremely eager to learn about them. I’m not an Amazon engineer but I’ve been bitten by oncall stuff.

I don't have any particular reading material, and my example of the on-call load at AWS that I'm referring to is probably very basic to most people.

On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis.

This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different.

It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.

There is something to be said for this approach. If the root cause is to be fixed someone needs to look at it in depth rather than running some play book procedure to recover. If you have too many problems though you're beyond the point where that helps. Let's say your software has worked flawlessly for a year, no issues, now an issue pops up, the engineers should definitely spend a lot of time understanding it, understanding why it popped up, fixing it properly and fixing the underlying process/org causes that made it pop up. It should not be "follow some playbook to recover". If issues pop up every week this is unsustainable, you're well beyond the point where stuff can actually be fixed. Automation has its own dangers, it is additional software to maintain, it has its own bugs etc. The right amount of automation makes life better for sure.

>If the root cause is to be fixed someone needs to look at it in depth

The root cause has already been looked at in depth 999 times when the same issue has come up. It's already been RCAed and the fix has been put in the backlog to be implemented sometime next year. In the meantime while we wait for the fix, we will continue to do a full, ad-hoc RCA every time the exact same issue appears, with the exact same results every time, because managers genuinely think it is a valuable way to spend our time.

I understand your point, but the relative utopia of a team you're describing is not really the situation I'm talking about. We have on-call periods where the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response, even though we already know beforehand what the root cause is and what the fix is. It's an incredible waste of time and contributes significantly to on-call engineers being overloaded, and yet we continue to do it and then are baffled when all of our engineers leave the team due to being overworked.

There's also nothing excluding runbooks and root cause analyses from existing together, either. In fact, most good runbooks specifically include steps to determine when an RCA is necessary and how to conduct one. There really is no excuse to not use runbooks as much as possible. If over-reliance on runbooks is having a negative impact due to engineers not applying personal judgement, then that is certainly an issue to be addressed, but the answer is almost never to completely abolish runbooks and documentation.

> the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response

Yeah, this sounds like a very bad situation where management won't let you do something that reduces ops pain because it isn't the most desirable solution, but they won't let you prioritize the right solution either. The next thing that happens is that on-call folks develop ad-hoc quasi-runbooks and share them amongst a subset of people (or just keep them to themselves to make their own life easier) and those quasi-runbooks become critical to ops, but not documented or shared by everyone. It's pure dysfunction.

This does sound pretty dysfunctional. You'd think that for something that's causing 999 on-calls getting the root cause fixed would be a priority. What I described obviously falls apart when the team has no ability to actually fix issues. Perhaps the original intent was to get those issues fixed but that somehow got lost as the org grows larger.

It is absolutely fascinating how wrong this approach is.

Every single issue that comes up in on-call should be evaluated under the lens of “does fixing this absolutely require human judgement, or can it be automated, ideally by fixing the code in the main system. If it does require human judgement, are there ways to redesign so that is no longer true?”

I'm not the parent but lemme chime in on this topic. It's pretty simple, if you don't build crappy software you won't get a heavy on-call load. You're really asking how to build great software. Build strong teams, with experienced people, follow good practices, reward quality and stability and not features or lines of code, reduce complexity, etc. etc. I've worked on software used by millions of people with a very low problem rate and then I worked on software used by hundreds of people where nothing ever works. Often in the latter the team, through lack of experience or ability, assumes that this is just the way all software is. There's plenty of examples of widely used software systems that are generally quite reliable and well built, and there's plenty of examples of stuff that's garbage, held together by duct tape, works by chance.

This is oversimplification.

One reality is more like "You are handed a system with a heavy on-call load. Make it better."

The other is "The system you built was great with 1e9 load, but now we're heading for 1e12 load. Make it better."

Sometimes throw it away and start again is the right answer

In my experience:

  1) Make a goal:  we should get paged for at most N incidents per week.  That goal should be closer to 0 than 10 IMO.
  2) Track stats on this goal, both in aggregate and broken down by cases you think you can address separately.  Example: tickets from alarms vs tickets filed by people.  Alarms having to do with external dependencies vs alarms caused by your own bugs.  Don't just make this a "it seems like we've had fewer pages lately" thing.  Real numbers.
  3) Review these stats on a graph every week. Someone should have an explanation of why they have spiked, why they haven't dropped, the breakdown of problem type, etc.  There should be congratulations when they drop and a request for plans when they don't.
  4) have management that can communicate upwards to leadership that ops improvement is a priority for your team and that you ultimately won't be able to continue other feature development if you are always mired in ops pain and people are either busy with mundanity or driven to leave the team.
  5) dedicate time in each sprint to working on the most recent identified target from plans made in #3.

This isn't particularly complicated, and typing it out almost sounds like I'm giving you worthless common sense advice, but I think the key here is that multiple levels of your organization need to commit to making this important enough to spend time reviewing it, agree it is a priority, and put actual dedicated dev work into it.

edit: formatted so indented list is readable on mobile. I have no idea how to do this without a code block. HN, please make this easier :)

There is a lot of SRE content about fixing on-call.

My short summary is, fixing on-call is a HUMAN problem, not a technical engineering problem.

Here’s an excellent place to start:

https://monitoring.love/articles/how-to-improve-on-call/

There is a wonderful list of talks and resources there, and those will lead you to yet more concepts and ideas to research.

> If there’s actually best-practices here that help that we’re not already doing...

Unless you have a leadership that recognizes the time sink on-call generates, all the best practices in the world are not going to help. This goes back to leadership culture. If they aren't in the trenches taking the same rotations with the on-call staff and at least simply staying up with them when the calls come in, then they will need metrics they believe to understand the deleterious effects of excessive on-call. This doesn't even begin to touch upon excessive on-call environments strongly signalling ignored technical and/or product debt that drags down new feature implementation and innovation. Leadership that truly comprehends this will allocate significant (20-25%) of time to addressing it, and push back on feature demands.

If you get to that point, then 80% of your struggle is over. Once you have leadership backing bringing the necessary time and resources to bear, reducing on-call is mostly lots of bookkeeping and documentation to identify trends and commonalities to prioritize and fix. A lot of the specific implementation details depend upon what you have available to leverage; ideally put someone in the PM role of tracking details for weekly stand-ups who is a relentless detailed note-taker and follow-upper.

The lack of investment into decreasing on-call pain is a real factor. I work on Oracle cloud (OCI) and at least some of the orgs (VP-down) have figured out that this is something worth focusing on, and the on-call gets better and better as a result. My original team had an average of something like 50 pager-worthy (sev2) events per week until we got moved into a new org that had the right philosophy and we relentlessly drove that down because management realized that engineers made miserable by mundane ops fake-emergencies would eventually get fed up and leave, and that's not what they wanted (afaik, OCI has no such forced attrition). So we got put on a program of relentlessly tracking and categorizing the sev2 counts and committing to improving those numbers over a period of time. 25% of dev sprints were dedicated to improving ops (tools, better alarms, fixing long-backlogged bugs that led to pages), and now that team's ops are pretty easy and they are free to work on new features, which everyone prefers. I've since moved to another team whose ops had already had this optimization done, and I've never experienced a bad week of on call there.

I won't pretend OCI is a panacea (lol google oracle cloud toxic work environment for latest stories) but at least they don't lack this particular piece of wisdom. The sheer number of regions they plan to operate doesn't really allow them to ignore dumb ops problems.

SDE1 here. IMMV of course but on my corner of the org I've seen a bunch of team members raising concerns regarding tech debt for the L7 to shut it down as it got in the way of delivering the features he wanted to deliver.

Also the elephant in the room is how the company relies on trial by fire as a form of performance evaluation, which involves inexperienced SDEs being pushed to deliver alone chunks of major projects in spite of lack of experience or insight.

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it.

Jeff Bezos can afford a private space program because - in part - his Amazon retail model is based on working people until they are mentally and physically broken, while paying them a pittance, discarding them, and moving onto the next group of people. He would rather spend money on crying booths and astroturf campaigns and buying newspapers than change that.

Why would he think about the expendable meat-units in AWS any differently?

Because We are knowledge workers, Not manual labourers!

I'm in non-AWS and things aren't any better here. It is very much dependent on the manager and team, and I've been optimistic (~3 years now), but it gets harder as time goes on.

> I'm just not 100% sure about the whole PIP scene. Our service was extremely critical and we were extremely understaffed. So I don't think it applied to anyone in our org but I know of other teams who would have no issues in taking in a fresh college grad, making them do work for 6-12 months and then just randomly putting them on PIP.

Our team is over-worked and has a large ticket queue, constant sev-2 pages, understaffed, etc. - and yet they still PIP'd (and then fired) someone last year who didn't deserve it IMO.

Amazon has a lot of money to waste. It is cheaper to hire someone so they can fire them in a few months to keep the rest of the staff scared enough to overwork themselves so they can understaff.

They play a good game.

> Amazon has a lot of money to waste. It is cheaper to hire someone so they can fire them [...]

So... Amazon has a lot of money therefore they have to resort to money-saving practices? Or is the causality the other way around? How do these two sentences fit together?

I don’t think they do, but it’s a way to smear and make all Amazon engineers unemployable by proxy. I already have difficulties finding interviews at good companies because the Amazon name implies IBM level talent instead of Google level talent.

Amazon has a lot money. These practices make them additional money. If they didn't have a lot of money they couldn't afford to hire to fire.

How can they do both? - They waste money when they hire to fire and constantly onboard. - They make money by not staffing enough resources but by using fear of being fired to force overtime.

Maybe the hire to fire costs are justified because the savings they get by understaffing outweights the cost.

It’s a “virtuous cycle.”

Wow that’s a huge wtf. You don’t have enough manpower, and then you take somebody who’s contributing and fire them? Seems totally insane.

Same. My team was down to 4 people and they put the L6 external hire in Focus (the informal coaching plan precursor to PIP) and he left. And we own critical services.

>but I know of other teams who would have no issues in taking in a fresh college grad, making them do work for 6-12 months and then just randomly putting them on PIP.

stack ranking came up in another thread a few days ago and this practice of forced attrition seems like just another way to do the same thing.

I thought it was understood that this kind of structure just incentivizes internecine fighting and politics over and above any imagined positive effects.

Mind-boggling that there still exists upper management that thinks otherwise.

I mean, early Amazon must have hired a lot of ex-Microsofties with stack-ranking PTSD, being based in the same area. You would think they would know better.

It seems like the more employees you have under you, the less you see them as human beings instead of inputs and outputs into a Rube Goldbergian machine you are trying to keep running.

> It seems like the more employees you have under you, the less you see them as human beings instead of inputs and outputs

I mean, there's a department in most companies called "human *resources*" to give away that employees are still resources, just human ones.

well said

> Mind-boggling that there still exists upper management that thinks otherwise.

Far too many Harvard MBAs calling the shots, I guess?

I know one Harvard MBA (also an engineer) and she was a brilliant leader of tech and teams.

What is really problematic is that from a management perspective they're doing really great: company is growing and extremely valueable, stock price is doing great, revenue and profits are doing great, and bezos is the richest person in the world.

Management has zero incentive to change any of problems you signal, and probably don't see them as an issue. Probably the opposite, they see this as a winning system, and who can blaim them. AWS practices are often used as examples of best practices: you build it, you run it, 2 pizza teams, api first, etc. Survivorship bias and all, the probably regard the other characterestics of the current system as best practices as well.

> Management has zero incentive to change any of problems you signal, and probably don't see them as an issue.

It sounds like management intentionally created these practices. This is from a recent NY Times article about Amazon warehouse workers, but it wouldn't surprise me if this attitude was applied to all workers in the company to some degree:

https://www.nytimes.com/2021/06/15/us/politics/amazon-wareho...:

> 5. Many of Amazon’s most contentious policies go back to Jeff Bezos’ original vision. Some of the practices that most frustrate employees — the short-term-employment model, with little opportunity for advancement, and the use of technology to hire, monitor and manage workers — come from Jeff Bezos, Amazon’s founder and chief executive.

> He believed that an entrenched work force created a “march to mediocrity,” said David Niekerk, a former long-serving vice president who built the company’s original human resources operations in the warehouses.

> Company data showed that most employees became less eager over time, he said, and Mr. Bezos believed that people were inherently lazy. “What he would say is that our nature as humans is to expend as little energy as possible to get what we want or need,” Mr. Niekerk said. That conviction was embedded throughout the business, from the ease of instant ordering to the pervasive use of data to get the most out of employees.

In many cases software engineers overestimate their own value to the company, often to the point were they'll ape the attitudes of owners and management (e.g. rejecting the the idea of a union out of hand). But the fact of the matter is they're cogs just like warehouse workers, and in many situations competent management will exploit the hell out of them to extract maximum value for the shareholders.

When did humane treatment, job security, and the worker's collective voice leave the picture? Are we to simply praise everything laid out here?

Some time in the 1980s, when people were convinced that if they stabbed themselves in the back, they had a shot at being millionaires.

Honestly ? When communist stopped being a real threat sometimes in 80s.

As horrific as communism was, the average worker in the west probably indirectly benefited from capitalist being scared of something similar happening closer to home, and gave concessions.

After the fall of communism that trend reversed and workers are steadily loosing ground.

As someone who was born in one of those communist countries I am glad that its dead, and I don't want it to ever come back, but It did serve as an example of what can happen if you try to squeeze too much out of people.

It's a pretty interesting take given that Bezos himself is the most entrenched person that Amazon has.

If this was truly the case, humans would have evolved very differently (and it's needs would be substantially smaller.

Also, what about all those people working for little gain?

We'll it might be true from Bezos's perspective, even if it's not true generally. Basically Bezo's goal is to get people working as hard as possible to enrich him and his shareholders. I think in many cases people expend more energy than needed to get what they want or need, but it's almost never in the service of getting some asshole billionaire they've never met a little richer. The energy people get from a novel job or from trying to prove themselves to a new community is short lived.

Also, as another data-point: slaveholders of all eras could not stop complaining about how lazy their slaves were. They expected the slaves to give their all, even though practically nothing was in it for them.

They are doing a great job from a management perspective but it is not good place on average for a developer. Both can be true.

Management only needs to change if all developers go else where.

> Management has zero incentive to change any of problems you signal, and probably don't see them as an issue.

SDE1 here, I might be completely wrong but I'm inclined to believe that upper management is kept practically blind about the real struggles and issues experienced on lower-levels, and this scenario is abused by managers to scapegoat their way out of problems they have to deal with, specially the ones they created. I'm not going to provide examples as they could be easily traced back, but I can say that I've witnessed a L7 overpromissing on a project in spite of callouts from lower levels, and once postponing the delivery of some milestones started to become inevitable then I've started hearing said manager frequently mention "firing offenses" on dailies.

You're completely wrong. Upper management is where the policies come from. They don't push back on URA targets because they either don't care or don't have the backbone to.

If your L7 is doing that, look to change teams. Incompetent people can't be fixed.

Exactly, big company works for Wall street not for their founder, employees and small investors.

worked in AWS for a while, but it was 5 years ago.

you’re right that your manager can make or break your experience.

For the first couple of years there it kinda sucked, mostly due to oncall (our ticket queue was at 3000 tickets at some point) and constantly being yelled at when things broke. We would basically only work on sev2s. Having the pager was super-stressful. We “owned” so much cruft/dead projects/experiments that a couple of time we were paged for something that we didn’t know existed.

After I build a little bit of leverage / gathered some political capital I somehow ended up in the position of “team lead” with I guess management’s intent to move me to be a full time manager (after the team’s manager was PIPed).

I made a good case that half the team is going to leave, me included, if we don’t do anything (6 people team).

So what did we do? I introduced a “secondary” oncall. After you were oncall for a week, you were secondary for a week. While secondary you got the chance to try fixing some shit without worrying about being paged every hour. You were also motivated because you just got offcall. People went for annoyances that would generate a lot of busywork or for… fixing the alerting and monitoring (a lot of autocuts due to wrongly setup alerting thresholds or even alerts that should not have been alerts in the first place). After we exhausted the low hanging fruit, we put some effort into automating some tedious task that would take a lot of time but understood and never meant to be done manually at the scale we did them.

Towards the end of the journey we aggressively deprecated/migrated the shit that was not used. By the end of this (took more than one year) we had an empty-ish oncall queue and for the first time in ages people coupd breathe (we now got a sev2 every other week - which in Amazon terms is freaking awesome).

I wish this story had a happy ending. There was close to zero recognition for what happened there and most of the team migrated together, internally, to another opportunity after. I left Amazon 6 months after this migration. From what I hear from the people that stayed there, entropy took over and in another 2 years they were roughly in the same shitty place as far as oncall goes.

I use reducing the rate of incoming tickets as the primary OKR for on-call engineers. Has never failed to reduce that burden over time. Just like you say - I'm convinced it's the only correct strategy to make the problem go away.

if you set a goal but don't empower people I think overall you end up killing morale. IMHO, reducing the rate of the incoming tickets can only be done with investments that are not fire-fighting. If you're doing that I think it's awesome.

Yeah, absolutely - that's the whole point. If the on call expectations aren't serving that goal specifically, we need to fix/change the on call expectations, so it's a moving target over time.

If there's too much firefighting, that means we owe additional support to the team or need to rebalance other work appropriately in the interim.

The opposite is also true. If we don't have too much firefighting or valuable de-risking to do, we can take on more product work.

This is a familiar story. Reducing on call load is actually generating a lot of value by freeing up developers and keeping them stress free. This fact is almost never recognized by product driven managements. Incredibly fucking annoying, considering that fixing such a system is not a trivial task.

I was disappointed to hear that it ultimately didn’t work out. A quiet oncall rotation requires good processes and a somewhat competent dev team.

Piggy-backing on the top-most-voted comment for those that will only read this one and not go to page 2 or onward.

Drama gets heavily upvoted.

If you keep scrolling you will find many individuals whose experience did NOT match this person (or OP's), who have a good work life balance, solve challenging tasks, and think this is the best job they've ever had.

I'm one of those people.

I've been at Amazon for almost 10 years, worked on a half dozen teams. Sure, there were outliers and a couple bad managers I've worked with, and some folks who left the company on bad terms.

But by and large, the experiences I had in 2 different countries on 2 different continents were mostly positive.

Yes, my job is tough. Yes, the oncall can be challenging sometimes too.

But I've also never stopped growing in the last decade, and have achieved bigger things than I ever thought possible.

Every single product and feature I've shipped have been immensely successful because of Amazon's deeply ingrained "Working backwards from the customer culture". And since that's what drives me - shipping cool stuff and having users on Twitter go crazy for it, I have never been more fulfilled.

Amazon is huge. There are now >60,000 Software Development Engineers worldwide. I'm not in Seattle but I'm in one of the largest offices outside of it.

So I'm sure every single horror story is true. But even so, they are a minute exception, and not most people's rule.

Why is it that amongst all the tech Co's, Amazon seems to have the highest proportion of horror stories? It's almost a meme at this point.

I mean the forced attrition and PIP culture is a known thing (x% of employees must be let go each year). Google/Apple/etc. do not have forced attrition or a known PIP culture.

Bezos is on the record for having said "humans are lazy and we need to give them incentives to quit".

No other Tech CEO has said that.

So perhaps your story is the anomaly?

I find it easy to believe that Amazon is a shit place to work (as a SWE) and that Amazon is a massive company with different bits being different. But I also find it easy to believe that a meme could just be wrong. It’s also a meme that Google has the best engineering and that their products get shut down after a year and that it sucks to be their customer. If everyone at Google is convinced at orientation that Google is some great company with great engineering then that can create a brand outside the company by the force of the size of their workforce. Maybe Amazon just has a different employer brand.

Don’t Netflix also hire relatively many employees also? Perhaps I’m just misremembering. And wasn’t stack ranking and the like quite popular say 20 years ago? It isn’t obvious that the Google or Facebook of Apple or Amazon way is the best way to manage a business. Lots of people online have complaints about each.

And I’m not even particularly put off by your Jeff Bezos quote: it doesn’t need to be a bad thing for someone to have an incentive to quit something they are not suited to as they could then go on to something they are better suited to, and it’s true that a lot of people (even well-paid with alternatives) will spend a long time going to a job they dislike because it is easier to stay with the status quo than to leap.

I still feel like the hire-to-fire stories are directionally right (and so are eg stories about shipping new products being the only way to get promotions at Google leading to warped incentives)

It’s also a meme that Google has the best engineering and that their products get shut down after a year and that it sucks to be their customer

To be fair there is a website dedicated to all the things killed by Google - https://killedbygoogle.com

No other tech co has a website like that, partly because it's incredibly difficult to get promoted at Google so you get a bunch of competing products that get spun up.

https://killedbymicrosoft.info/

You have the cause and effect backwards. Memes motivate people to make websites. Also, other large tech companies have shutdown websites as the meme has spread from Google.

Even in my charitable view of my company, Amazon is INTENSE.

We set extremely ambitious goals, and orgs are only expected to meet 70-80% of them.

It's an environment that indexes extremely high on autonomy and flexibility for each person and each team.

It is without a doubt a sink-or-swim environment. But, by and large, the ability to swim is based on individual merit.

Totally agree. My experience is my own and definitely not reflective of everyone else who works at Amazon. They employ tens of thousands of engineers and many of them are happy. I just wanted to share my story - not trying to generalize this for all of Amazon.

But I wouldn't say it's a minute exception with so much of it being revealed. For every story that's revealed you can assume that there are others in a similar situation who do not publicly share their story.

> If you keep scrolling you will find many individuals whose experience did NOT match this person

So I read this comment some time ago then scrolled quite far down the comments.

I have to report that there were very few people not expressing similar experiences than OP. It seems it is predominately resentment from people claiming to have worked for Amazon all the way down.

It's called Survivorship bias. Before coming here, I too thought the same. That the stories were exaggerated, that those experiences were outliers and a minority. But then I saw my team's L6 SDE get put on Focus within a year of joining and leave. If they had caused Sev1s or broken something critical, then perhaps justified. But no such thing.

This place puts you in the middle of the ocean when you don't know how to swim. You either sink or swim.

> If they had caused Sev1s or broken something critical, then perhaps justified.

That's not the metric for success or failure. It's delivering results.

Except there are 14 LPs and nobody says any one LP is superior to the rest. If you need to consider "Deliver Results" supreme irrespective of anything else, then it needs to be made clear. Just do away with the rest of the LPs.

This jives with what I observed as an intern some on an AWS team some years ago. The oncall rotations seemed absolutely brutal and the engineers were so busy and stressed out fighting fires that they barely noticed my existence (which I was okay with) and the tech debt kept accumulating between because between new feature launches and firefighting there wasn't much scope for anything else.

My intern project was a fairly no brainer tech debt item that automated a lot of the deployment process and saved our lead engineer several hours a week in babysitting deploys. I resolved to never work on a cloud infra team after that -- while the internship was fine, being a full time engineer seemed absolutely miserable.

I don’t get this. In this market, if you’re good, why take a job with any on-call work? It’s thankless, shitty work. If you can deliver features, that’s good enough to get paid extremely well at other Amazon-scale companies with dedicated SRE teams. What’s making you stay?

Because changing jobs is really fucking difficult.

I'm probably one of the biggest proponents of "quit your job, you deserve better" that you'll ever find, but even I have to admit that finding a new job is ridiculously hard. Even in "this market", even if you're a top engineer, it's still ridiculously hard to even get an interview, let alone get hired.

There is only a limited amount of companies that will pay at the same level as Amazon, and those companies often have months-long interview processes with ridiculous requirements that, even if you are a top engineer, still require a lot of time and effort be set aside to prepare for the specific interview processes that the new company is looking for. And that's to say nothing of the nebulous "culture fit" that is just as likely to prevent you from getting an offer and is completely unaffected by how "good" of an engineer you are.

Almost everyone I know at AWS is interested in switching jobs/companies, but it really is not just something you wake up one morning and decide to do. It's a long, perpetual process that can take up a huge amount of time and effort (stuff you don't have a lot of when you're working on-call at AWS anyway), not to mention has huge implications if you are relying on your job for something like a work visa.

It sounds like the real reason is that the money is worth it.

The money is probably the thing that keeps most people in the job, but it's not like willingness to take a pay cut magically makes jobs fall into your lap, either. It expands the pool of companies you could possibly work for, sure, but everything else above that I said still applies.

There's also rarely a guarantee that the new company you're trying to join is significantly better, and companies intentionally make it difficult to get the inside scoop on work-life balance/on-call responsibilities until after you're hired. Personally, I would love to find a new company to join... but my experience interviewing is that it takes months of effort for a potential pay cut and no guarantee that I won't just end up in another shitty situation and paid less, so I struggle deciding if it's actually worth it.

> not to mention has huge implications if you are relying on your job for something like a work visa.

Isn't it just apply for a new job and as part of the hiring process they handle the paperwork? I know people on work visas that have jumped jobs every two years.

There are many, many companies that will pay at the same level as Amazon and hire you after less than ten hours total of interviews, as long as you think of compensation the correct way: in hourly terms

The answer is 1) many people at Amazon like me have inferior intellects and everyone knows it, so it’s harder to get a better job and 2) there are many services that don’t have dozens of sev2s per day because ultimately you do own what you build. It’s a big reason why Lambda is used to heavily internally now - we cut out a class of operational scaling issues out by design.

Also about a third of Amazon's engineers are on a visa. You can transfer jobs but it is never smooth sailing and full of gotchas.

Yeah there's no way I'd put up with this. Trillion dollar company and still operating like it's amateur hour on many teams it seems.

The org is getting what it wants out of the team; why would they fix it?

As long as you've got meat for the grinder, you can keep making sausage.

Also: the cruelty is the point.

I'd disagree with that.

One doesn't need to be cruel to be callous or oblivious.

One can intentionally remain oblivious, but that doesn't imply that the cruelty is an actual goal.

See also people who happily eat meat (myself included) who are still troubled (as a 5th order concern) by the process.

American capitalism is built on employee heroics, but at least you can get the AWS stamp on your resie, which, let's be honest, will make you the "it" girl of job hunting.

I think you overestimate the appeal of a handful of large companies, taken by itself, on a resume if only because they come with downsides as well.

The level of abuse and mismanagement I put up with before I got older is pretty shocking. Young people have a much higher tolerance for BS, in many cases because they don't have enough experience to know that their workplace is broken.

Or worse, they may internalize it as normal and unreflexively continue breeding the problem. Those olderish managers of young people were young people themselves, once.

Yeah, it's much easier to end up in a malfunctioning workplace than a good one. The number of healthy workplaces where everything seems to just work is shockingly small, and you are actually lucky to end up in ONE through an entire career. And then you can never go back.

This works when the company is hot, but eventually it's not. When the last time you saw HP, or IBM positively on a resume?

Big companies breed big company problems that can leave engineers poorly equipped for the rapid delivery world of startups/scaleups.

"HP" or "IBM" carry a lot more weight on a resume than something like "BF Consulting Ltd"

As someone with one of those two on his resume, I'd say you are not wrong.

That said, it's a bit more nuanced. It still looks good if you can share experiences that prove why it's good.

Ex. "I learned what it's like to be closer to the cutting edge in some respects, but I also learned how that should be secondary to delivering a good product and good customer service due to issues X, Y, Z observed. Also, using technology A is great, but it might not be worth your investment at current stage."

This kind of statement illustrates that you learned multiple things of value, and hopefully avoided bad habits and are pragmatic and worth having. It's possible they can leverage your experience to avoid making mistakes in the next growth stage.

So, yes, it doesn't always look good immediately. It's up to you to prove to a questioner why it was good and hopefully they also agree.

"Big companies breed big company problems that can leave engineers poorly equipped for the rapid delivery world of startups/scaleups. "

Why are big company employees not suited (or trained for?) to startup like environments?

There are several reasons big company experience doesn't translate.

1. Legacy tech, "cool" big companies have the latest stacks but over time these stacks become old and the dev practices atrophy. The biggest cases of this I've seen are engineers unable/unwilling to do their own QA as its "not their job". Or unwilling to adapt to new technology the startup may be using.

2. Politics: Big tech companies have major politics leading to pathological "not our problem" conditions. Efficient and successful startups/scale-ups need to minimize politics, and some employees from big tech companies will have found this to be one of their primary skills.

3. Ambiguity/Timeline constraints/dealing with crap: At a big tech company everyone is expected to be at the top of their game and the company rarely faces deadlines that are not self-imposed. An engineer may expect 100% test coverage, crystal clear product requirements, and no risk of failure. Dealing with sub-optimal conditions is common in startups/high growth companies.

4. Definition of success: A big company may strongly value marginal contributions as prized wins. Shaving ms off of a frequent call can drive real monetary improvements when the company has hundreds of millions of customers. Making engineers marginally more productive has huge benefits when you have 50k+ engineers. Startups often just don't care about these things and simply won't value the skills necessary for this work in most cases.

On the flip side there are great engineers/managers who learned what made the big company successful and how to navigate internal obstacles. These employees are likely to be gold to a startup, but they are also gold to a well capitalized company with vastly more data on just how effective they are than a startup has. Odds are, startups are interviewing/hiring the employees the big companies don't care about - something the hiring firm often implicitly knows.

but what's the point?

Don’t conflate corporate size with engineering competency. It’s counter-intuitive, but oh so common to see this. That’s why so many engineers want to work for start ups!

> Don’t conflate corporate size with engineering competency. It’s counter-intuitive, but oh so common to see this. That’s why so many engineers want to work for start ups!

I thought start ups were all about letting tech debt pile up like there's no tomorrow, since there might literally be no tomorrow for them.

The truth of the matter is that I've never seen anywhere where this isn't the case to some degree. Long term thinking is rare.

I think it's very safe to assume the level of technical rigor in a given undertaking just falls to the minimum required unless there's a very strong force keeping it in a higher state. Maybe places like NASA JPL or Apple manage to float above the minimum because of a really unified and powerful culture, but outside of that I'm thinking it's more or less universal. e.g. the 737 MAX debacle illustrates Boeing's stochastic search for the lower bound of technical rigor when it comes to flight control software quality.

> I think it's very safe to assume the level of technical rigor in a given undertaking just falls to the minimum required unless there's a very strong force keeping it in a higher state.

It really depends on who's setting the tone. If it's an owner or manager taking an active interest, I think your observation is true.

> Maybe places like NASA JPL or Apple manage to float above the minimum because of a really unified and powerful culture, but outside of that I'm thinking it's more or less universal. e.g. the 737 MAX debacle illustrates Boeing's stochastic search for the lower bound of technical rigor when it comes to flight control software quality.

IIRC, Boeing has made a years-long effort to cripple their unionized engineering workforce. I don't remember exactly where I read this, but for a long time they had a very effective, rigorous organization, but management (from McDonnell Douglas. IIRC) make a lot of changes that messed it up.

Bloomberg Quicktake (which is one of the few news outlet Youtube channels I recommend) did a good 20 minutes piece on Boeing's internal cultural demise [1].

[1] https://www.youtube.com/watch?v=EESYomdoeCs

I had an experience at a startup where the team was mostly experienced people who had been at larger companies and didn't tend to cut corners. It was a pretty good balance. We focused on doing the job well but didn't have big company meeting culture, etc.

I wouldn't want to work at a place like you described.

I think this is because Amazon loves to churn teams.

My team, not Amazon, had alot of churn when I joined. Basically a full turn over within a year. We lost alot of institutional knowledge and had to reverse engineer stuff all over the place.

(Disclaimer: Was with Amazon for ~7 years a long time ago).

I've been a customer of AWS across multiple startups and I've seen the overall quality of their products continuously degrade which complements your experience.

While they continue to launch new products at a rapid clip you can see small cracks beginning to appear as the products age. A permission issue that's not documented, a cryptic error message etc., They aren't show stoppers on their own but if you use AWS long enough you will be worn down by the cumulative pain.

I'm always surprised why industry leads (like Amazon) sometimes treat their products as an amateur treats his/her weekend projects. This is definitely not the worst as I know one of the leading options marketmaker has been using a giant shit mountain of MS Access/Excel VBA code to run their system since the 90s. Last time I heard about it (a few years ago) they are planning to replace that shit mountain with something new but I don't know if it's done now.

It's built into the culture to ship, ship, ship. Shipped code is better than good code, or clean code, or fast code. At my last job, the C-level was fascinated by Amazon success stories. They wanted to achieve the same success, so they urged us to ship, ship, ship. Unfortunately, we were all very seasoned engineers, and we knew the nightmare that would ensue if we purposely piled on the tech debt. The part of this article that refers to "every ticket taking 4 to 5 days regardless of complexity" should be a warning sign to ANYONE attempting to model their startup after Amazon.

The message should be: You are not Amazon, and you will NOT get to Amazon scale by modeling their worst practices.

I think this culture is OK or even good to a company in start-up because you have to be quick. Once it grow into maturity those rules should be abandonned.

There are a few problems with that approach. One is that there is rarely a good time to say, "ok, we've proven that this idea works, let's now go back and do it correctly". It's a constant stream of fixes on a system that "already works". Secondly, telling management that a team was able to go fast previously, but is now going to start being slow so that things can be done correctly, is a quick way to be shown the door. This is evidenced by Amazon and that thousand line Java method that's probably existed for 10 years. Finding that time where you're mature enough to switch gears never seems to happen in practice.

I now advocate a "cut corners, cut scope, but do NOT cut quality" approach. Unit tests do not take much longer to write - but they pay dividends when it's time to refactor. I'm now back in a startup where the code was written with that "just ship it" attitude. The code is so terrible that it can take days to fix a bug. I can rewrite entire features (correctly) in that amount of time.

Shipping tech debt should be compared to the realistic alternative which is never shipping at all. The solution is not to ship slower, but to attract a better team and then retain them. They say the CEO's #1 job is recruiting and this is why. Actually more important is growing your revenue faster, if revenue compounds faster than debt then you're good!

I mean you can also ship a lil slower.

Startup founders and management types are obsessed with optimizing dev time at a day/hour granularity though in my personal experience, so it's a lost cause.

Can you really ship slower?

In the end it’s a winner takes all market, and AWS needs to out-innovate azure to win.

And as customers of AWS we’re all looking forward to the next re:invent for new features, and we’re voting with our money buying huge amounts of AWS services.

Also, as a potential employee, I would always sign with the company that has positive cashflow through customers that pay for shipped value, rather than a company with a great code base, but a bad sales track record.

Apparently it works though. Since both examples are described as "industry leads".

> This is definitely not the worst as I know one of the leading options marketmaker has been using a giant shit mountain of MS Access/Excel VBA code to run their system since the 90s.

Only one? :)

What I don’t get about Amazon is that AWS customers have way less oncall load, particularly Netflix, whose engineers could afford oncall 24x7 for weeks or months with perfectly normal work-life balance. Why can’t Amazon, the pioneer of cloud computing, achieve the same level of effectiveness?

Maybe I'm misunderstanding you but AWS customer's eningeers have less oncall load because the AWS engineers are doing so much of the difficult and failure-prone work.

That's largely what you're buying when paying for cloud services.

What I meant was that Netflix built their platforms and services on top of AWS services, just like AWS teams. Yet AWS teams have brutal oncalls, while Netflix teams enjoy great work-life balance.

Company culture and values flow from the top

But that’s because the AWS teams are presumably shouldering the burden, then.

I’m sure you’re right for some services, especially infra ones, such as EC2. Some other services, though, should be built on top of EC2, EBS, Lambda, S3, and etc, in which case Netflix and AWS teams use the same infra, yet Netflix internal services require much less oncall

No doubt Netflix is a much better run engineering org.

Its interesting that Netflix built their systems in a way that assumes the underlying platform is unstable. With Chaos Monkey and other systems they made sure things are resilient to flakey behaviour.

Had a relatively similar experience (finally understood what drinking out of a firehose meant), the silver lining of the experience though is it finally dispelled any notion of imposter syndrome when I realized everyone was running around as much of a headless chicken as myself ahah

> We had a lot of little scripts here and there which would solve extremely specific situations but no focus was ever put on in building a general framework or trying to reduce the ticket count.

wow, honestly this is surprising. For me as an end customer I was always impressed with the way the services are engineered. Kudos to people like you for making this happen. But then I was also under the impression that AWS has very good best practices to take care of repeating issues.

Wouldn't interim patch-ups cause stability issues in the long term?

> Wouldn't interim patch-ups cause stability issues in the long term?

No matter what people tell you or put on their marketing blog, it feels like this is the state of play for 99% of software teams. The only time it doesn't end up like this is when you leave time up front to pay down technical debt (like Intel's tick-tock model), and almost no one does that.

my senior once said to me that the world is held together by two things.

Ducttape and shellscripts. This was a while ago, but the older i get the more i tend to agree.

Very weird if you do get called out do you not take TOIL? and get paid on call allowances? Having this cost exposed will lead to things getting fixed.

This solution works based on knowing developers on CSS the huge mainframe billing system and my own experience as the lead for a smaller BT billing system I worked on.

And TBH a 1-2 kloc method isn't that big I have seen individual SPROCS over 2.5k

> I can safely say that it's totally up to your manager and pre-existing conditions which make up the job

Isn't it like that at any job?

So many of my FANG friends talk about the horrid code quality and sometimes terrible processes that are in place that I’m always surprised. Considering the amount of gatekeeping

how many months of work does it take to get the "stamp"? I'm assuming employers would be wary if it's only 6-12 months

If you're brand new to the industry I'd say stay for at least 2 years. I worked on a service which operates an enormous scale and while I'm thankful for the opportunity it takes several months to ramp up (I'd say about 6-8 months for my team). But obviously YMMV.

> Code quality was atrocious. We had one enormous Java method (>1000 lines) which would take care of nearly every single request coming into our service... With only about 7-8 unit tests

AWS is pretty reliable for the most part so I am pretty surprise that the code quality is that bad.

I think one thing I learned from AWS is that there's so much hidden away from the customer. There definitely were (and probably still are) many issues which the customers won't actively experience. Reliability doesn't necessarily equate to good standards and good practice.

But yes, from a customer point of view, AWS is pretty nice.

Reminds me of that old quote by John Godfrey Saxe:

”Laws, like sausages, cease to inspire respect in proportion as we know how they are made.”

"I just had one for breakfast." -- Jim Hacker

I think that highly depends on the service. The new App Runner service for instance is a wild ride of buggyness, lack of testing and incorrect documentation.

> AWS is pretty reliable for the most part so I am pretty surprise that the code quality is that bad.

I'm not totally surprised because of two factors: very stable product definitions and lots and lots of users.

A number of years back, I was talking with people at a famous and popular site with a broad audience. I asked them how much unit testing they did. They said that particular isolated pieces sometimes had tests. But most of the user-facing stuff didn't because they had one-button rollout and one-button rollback. Instead of bothering with unit tests, they'd just frequently release changes, watch the metrics and the customer support queue, and quickly roll back if they'd introduced a bug.

For very, very popular services, a second of being live will exercise more code paths and edge cases than even the most dedicated testing team could ever dream of.

We hear a hell of a lot about testing but the most fundamental piece of software quality nowadays is the release strategy: running on tee'd live production traffic, canarying, metrics and alerting, quick roll backs, etc.

Depends on if you are serving ads over cat pictures or routing air traffic. Different solutions for different problems.

It's a very short-sighted view on testing, although I'm not surprised SREs would say it. The biggest problem with software deployment is that it is owned and managed by people who have no vested interest in developer productivity, including devops engineers.

A major goal of any org should be developer productivity; otherwise you are just hemorrhaging money and talent. When I say developer productivity, I mean: How confidently and quickly can I make a shippable, rollback-free change to a unit of software?

If you are the dos equis man of testing, "I don't always test my code, but when I do, I do it in production", then you can't confidently make any change without risking a production outage, so you play lots of games, like you mentioned, around canarying, rolling out to a small percentage of users, etc., but at the end of the day your developer productivity has absolutely tanked.

The goal of any system maintenance should be that a developer can quickly make and test a change locally and be highly confident that the change is correct. The canarying, phased rollouts, and other such systems should not be the primary means of testing code correctness.

Yeah, I really appreciate excellent rollout strategies, although I suspect a lot of them are more developed out of self defense by SRE teams. I see it as a series of safety nets: I'm still going to write tests for my code so that I don't have far to fall if I make a mistake. But I also want a safe rollout so if I miss the first net I don't splatter on the pavement.

And I totally agree with out about developer productivity. It's just not a consideration in most places. For example, in a factory or a restaurant, meetings are things that happen rarely and in constrained time slots, because everybody realizes that production is primary. But in most software companies, actually getting work done is second priority to meetings.

Agreed. I was an SRE for over a year and the philosophy is that anything that is shipped can be broken. SRE is all about detecting, limiting, and mitigating damage. I think this is the right philosophy for SREs but should not be the total picture in the org.

I am also agreed in that I anecdotally often see a disregard for automated testing. I am still trying to understand how to eliminate this tendency. I know that in every software project I've had a major hand in building, I've helped ensure automated testing, with a heavy emphasis on unit testing, become a major part of team culture, and I've always felt the tests more than paid for themselves over time, even in the relative short-term.

For sure. I'm still trying to understand it too. Two things that have helped me:

To contain time pressure, I like a kanban board with small units of work. If the team has a history of steady delivery of small lumps of useful stuff, managers are more willing to trust that we know what we're doing.

To mitigate the natural human lack of humility, I start with the rule that every bug requires a test before fixing. People may act as if they won't make mistakes, but it's much harder to claim that when we have a live bug. And then I like to introduce pair programming. Collaboratively adding failing tests and fixing them (as with ping-pong pairing) makes it fun.

What approaches do you use to shift things over?

I find it's much easier for greenfield projects than existing projects because you can lead by example. Also, with these projects there are opportunities to define team rules and culture for the project from the beginning.

Perhaps the greatest inertial force against improving test automation is the truth that any change to a in-production product incurs risk, coupled with the fact that adding tests often requires refactoring to make the code more testable. Techniques like writing tests as you refactor never get employed because there is resistance to any refactoring occurring at all in the first place. The general principle followed is: "make the smallest change possible to accomplish the objective".

I feel like overcoming these forces requires bravery from management or the team, coupled with a longer term vision for the project. For some projects, which are limping along on life support, it may not even be worth it to improve code quality. However, for most other projects I believe there is a better balance to be had between not unnecessarily breaking the product and not being afraid to make relatively risky changes to improve maintainability of the product.

Makes total sense. I do most of my work on greenfield projects for that reason.

But I think you're right. There's really no low-risk path when you have a poorly tested code base. You can keep letting productivity decline, which guarantees eventual project failure. You can do a giant rewrite, which is hugely risky. Or you can gradually dig yourself out.

There's still risk there, of course. But it's in smaller, more manageable lumps. It seems like the clearly superior path to me.

If the release/rollback process is fast enough, and your detection of anomalies is fast enough, you can still have great productivity, and few relevant outages, when testing in production. Hell, there are situations where testing outside of production is never going to cut it, as generation of sufficient load of the right shape would take you a whole lot more of engineering time than the consequences of failure.

That said, the tradeoffs are different for different companies, and different services in the same company: Within the same team at $large_company, I owned code where testing in production, via deployments and an amazing feature flag system, was better than unit tests, while there were other areas where the build system would dedicate many CPU-hours to testing before any release. To be able to have that flexibility though, you need to know your systems, know your problems, and have great tooling for both testing in production and extremely parallelized test suites. Small and medium sized companies might not have either alternative, and we had both!

So what I'd say is that any general rule on what should be the primary means of testing code correctness is going to not lead to optimal productivity, and even more so if you don't have top quality of tooling across every possible dimension. It's perfectly OK to argue about specific examples, but without judgement of this kinds of things without having the entire story of what's there is just hubris.

I unfortunately can't agree with that sentiment. If I have to rely on production traffic to test my feature, then, at minimum, my feedback cycle is:

1) Make change. Think really hard about it to make sure it's correct.

2) put out code review.

3) get approval. Merge change.

4) CI pipeline builds and deploys to prod.

5) absent of alerts must mean it works?

Even if you have no QA environment and nothing between you and prod, I've rarely seen deployment to prod take less than 30 minutes. That's an hour feedback cycle. Contrast with:

1) write code.

2) write unit tests.

3) run tests locally.

The feedback cycle, especially when you get iterative, can get as low as single digit seconds. I run my tests and see a bug. I fix the bug, then re-run the tests. Similarly, for a more complex feature, I can break the feature down into multiple cycles of build, test, verify.

And that's not even accounting for the overhead of managing feature flags, which is not free. In the best case, you need to at least release a second PR to remove the feature flag when the feature is successful. At my previous employer, this step was often forgotten and resulted in real, consequential technical debt as it became harder to figure out how the product behaved based on which feature flags were turned off or on.

If you have an experience leads you to believe that production testing can more productive than local automated testing, I at least have never seen it occur in and I find it difficult to even imagine it being true.

That's an overly general statement. Can you do that for front-end code that stores all of its state elsewhere? Sure. Can you do it for a storage system? Absolutely freaking not. If you introduce a bug that loses or corrupts data, there's no going back. You will have committed the worst sin that somebody in that specialty can commit. Better to test as much as you can, at every level. Other kinds of code are often somewhere in between.

Also, even if it's true that being live will exercise more edge cases etc., it's a terrible way to test changes during early development. For one thing, there's no isolation. It becomes harder to determine which of several recent changes caused a problem, and that burden unfairly falls on the person who's on call instead of the person who introduced the error. And decent unit/functional tests allow "dumb" mistakes (we all make them) to be caught earlier than waiting in a deploy queue, allowing faster iteration. "Most recent change probably caused the problem" is a very useful heuristic, but the more low-assurance changes you allow in the less useful it becomes.

To drive the point home even further: I have found data-loss bugs in focused testing that didn't show up in prod for months. I know because in many cases I was able to add logging for the preconditions when I fixed the bug. No logs for months, then some completely unrelated and completely valid change by another engineer tickles the preconditions and BAM. That would have been an absolute nightmare for other members of my team, possibly even after I was gone. Based on those experiences, I will never believe that foregoing systematic early tests can be valid. The systems most of us work on are too complex for that.

"Test in prod" only works for trivial code and/or trivial teams. Not in the grown-up world.

Everyone should be testing in prod, in that you release code and see metrics and monitoring to show that everything is working.

Testing in production is not going “let’s see if this will work” it is “we will release and validate that everything is working as expected”

People need to get over the old school cowboys who jump on prod to see if something works.

Yes, everyone should release code and watch metrics etc. but I think that's at the very edge of what "testing" encompasses. Between model checking, traditional forms of testing, and shadow-traffic testing (which can test higher per-server load than prod), finding something after deploy should be like a parachute failure. Yes those happen, yes there should be a reserve, but if it happens more than once in a blue moon you have a process problem somewhere (quite likely between teams/services but still).

Tangential, where can I learn more about shadow-traffic testing? Books, blogs, tools etc.

Cindy Sridharan often writes well about these topics.

https://copyconstruct.medium.com/testing-in-production-the-s...

For this one I'd start about half way down, under the heading

"Shadowing (also known as Dark Traffic Testing or Mirroring)"

Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam.

Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy.

If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.

Teeing/dark launch/dual write strategies solve most of the issue for databases. Sure you run into concerns when changing the framework that manages that, but that's usually a far smaller surface area than your entire storage layer.

That said, you should have tests anyway.

Tell that to the millisecond of testing in production that makes the MRI fry the patient's brain, to the one that trades one trillion instead of one thousand naked puts, to the nuclear armaggedon launch check that canaries humanity...

>For very, very popular services, a second of being live will exercise more code paths and edge cases than even the most dedicated testing team could ever dream of.

Most of the code we care about is to handle anomalous situations. That AZ going down a week or two is a good example. It's when stuff like that happens that a bunch of code springs to life to keep things running. And indeed, things didn't exactly roll over just fine for us.

> AWS is pretty reliable for the most part

In telecom or traditional mainframes, for example, the compute unit itself was expected to be reliable. Individual elements of AWS are not pretty reliable in that context. Check out the single host EC2 SLA.

However, today most large or even medium scale software assumes unreliable individual elements and has redundancy at the program level. For that purpose, AWS core services are pretty reliable.

Way back I used to work in telecommunications at a place that provided POTS service. They are two very completely different worlds. Software engineers act as if 5 9's is a badge of honor, when really it isn't. When you are responsible for something that people use to dial 911 and can make the difference between life and death a few minutes of downtime doesn't cut it.

Right, and even five nines would be impressive compared to:

AWS will use commercially reasonable efforts to ensure that each individual Amazon EC2 instance (“Single EC2 Instance”) has an Hourly Uptime Percentage of at least 90% of the time in which that Single EC2 Instance is deployed during each clock hour (the “Hourly Commitment”). In the event any Single EC2 Instance does not meet the Hourly Commitment, you will not be charged for that instance hour of Single EC2 Instance usage.

This essentially forces the use of distributed computing for even small businesses.

EC2 is absolutely not meant for this, though. Use an abstraction layer like Heroku if you're going to not understand what you're getting into.

The amount of times I've had to 'advise' small businesses that are somehow running their small business site off a single EC2 instance's ephemeral boot volume is atrocious.

I don’t have any experience with huroku but what most small businesses need is a (perhaps simulated) reliable box on a fast network. As glorious as the paxos based present is, it’s overkill to the point of distraction for most businesses. The whole attraction of the cloud for them is not needing to hire sysadmins. Replacing that requirement with needing a devops team is even worse.

For something heavily used, like the EC2 and load-balancing, perhaps, but I am still experiencing PTSD from my last CloudFormation encounter.

AWS has the advantage of having so many engineers behind the scenes available for firefighting with a culture of pushing more than is reasonable that as a customer, that would sort of disappear. They simply have to occasionally make trade offs between unrealistic development/feature request goals and firefighting whenever the firefighting is needed. This also acts as another form of pressure to work even more to meet timeline goals.

Don't let your developers know that you're expecting them to always be behind, infinitely queued up with work, and constantly in emergency mode and they won't have much time to think about what's really going on and how efficiency is being pushed at the cost of their sanity.

This is indeed surprising. Any time we have slowness issues the usual recommendation would be to throw resources at the problem; increase cpu, add more memory et al. We used to lament that we should spend time debugging the problem and fix the actual issue. We then used to say that probably at places like AWS and the other biggies they'd be following some excellent best practices and we should also strive to reach that level of excellence.

AWS is very big, culturally, on making sure that all the bugginess from shitty code is not shown externally to the customer. Externally it might look like everything is fine to you, but internally AWS is a massive, leaky cargo ship with thousands of engineers running around 24/7 with duct tape and band-aids to plug the leaks.

Reply on news.ycombinator.com