Hacker News Reader

Comment by codingwagie | Hacker News Reader

I'm a long time Amazonian. The big problem is legacy employees run every part of the company. Almost any manager of managers has been at Amazon a long time, in that same job for a long time. There is no upward mobility at the company, unless you have been in some org 5+ years. In Alexa, the people running the core ML teams have been in Alexa since it started. Most people in decision making positions just got there first (10-15 years ago)

The software engineering paradigms used within the company create brittle rube goldberg machines of events flowing everywhere in the company. Almost all of them are on maintenance mode, where the oncall burns out the engineers and prevents them from creating new products. There is no knowledge sharing between team members. Legacy team members guard their technical platform knowledge to solidify their place on the team.

The engineers themselves are not students of computer science, but just crunch out tickets.

If Amazon wants to change they need to remove a significant amount of tenured employees, and actually promote young engineers into decision making positions.

AWS hasnt released an innovative product in a really long time

View on HN

> There is no upward mobility at the company, unless you have been in some org 5+ years

Sometimes I try to talk to my 83 year old dad, who was a Teamster at the same company for his entire 35 year career, about the software industry. He's so surprised how often people like me change jobs, how we switch companies as a way to get a raise, and just generally how different expectations are. When I told him about how I'd left a company partly because they didn't promote me after 18 months, he didn't say anything, but I knew what he was thinking. In his world, 18 months just isn't long enough to feel entitled to a promotion. Promotion is primarily seniority-based, and the company rewarded loyalty as much as, err, value creation. It's a different world we're in, but I'm not sure exactly what makes it different: is it the 21st century, something fundamental about the industry, or the fact that software people feel they have more mobility, and thus less loyalty and patience?

One thing I'm fairly certain about is that companies don't treat us worse than they treated blue collar workers in the 70s. I think we'd all be surprised by the poor selection of candy in the catered employee cafeteria they had over at the shipping depot.

There are a lot of factors for the change in loyalty but I think the single biggest one is from when the managerial class went all-in on outsourcing. Those execs didn't care that they had a factory full of workers that had been loyal for decades, they happily sent those job oversees for a bump in that year's profit (and the resulting bonus). The people making those decisions didn't care that so many families, towns and cities were ruined by these actions, through no fault of the workers.

Given that betrayal of the old social contract, why on earth would anyone choose to be loyal to such companies? Work loyalty still exists with individual managers or owners of small businesses that actually interact with their workers, but these days most workers correctly assume that most corporations are being run with little regard to anything aside from profit; and those workers act accordingly.

Loyalty died because the C-suite are comically greedy idiots and don't care to keep the good workers by recognizing their worth. Mind you, they never really did, but people just realize it a bit more now, plus things have gotten worse in that regard.

I can't tell you the amount of good colleagues I've seen leave because management was too stubborn to give them a 7% raise. The most baffling thing is that they'll have no qualms (or perhaps no choice) to subsequently replace them with someone else for at minimum double of whatever 7% would've cost them. This causes a cascading issue where people are annoyed that new hires get such a big boost while the current staff have to fight for every measly percentage increase, so the older employees slowly trickle out.

I've had it happen myself too. A place I really enjoyed working at, I found the work great and had a great relationship... But the CTO stubbornly refused to give me a 9% raise. I referred my friend and started looking elsewhere. I got a roughly 30% paybump, and my friend got hired to replace me for roughly 20% more than what I asked for.

So why on earth would I be loyal? The people running the show are hilariously short-sighted and can only see the beacon emanating from their VC buddies shoveling money their way, so whatever, I don't care ultimately if something I worked on dies or not.

Yes C-suite is very greedy so they let people leave instead of giving them a 7% raise, and turn around and hire someone for a minimum of 14% more.

They are either greedy or stupid. You have to pick one.

Why do you think greed and stupidity are mutually exclusive?

Because to be successful being greedy you have to be smart. Like on a long term basis. You can't make big stuff without a smart strategy.

"keep flogging the same shitty technology while bribing gub'mnt officials to ban new stuff" is not especially smart. no need for an advanced degree for that.

but boy howdy does it work, and make a lot of money.

> Because to be successful being greedy you have to be smart

There are plenty of people who are greedy and not smart. I'm not sure what basis this statement has. Plenty are greedy and not smart for their entire lives.

This just perpetuates the bullshit narrative.

They are not smart. For many stupid would be an understatement.

The amount of stupid people who fail upwards is far larger than the few and far between who get there on merit.

My direct boss at the moment is one of the smartest people I've ever met. The people running the plant are some of the stupidest I've ever seen. Constant mistakes, short-sightedness, asking questions anyone who works at the plant should know in their first week, etc.

Best part is it's a relatively small city, so everyone knows everyone kinda. Our plant manager (top guy on site) was in a middle management role at his old plant (the worst in the city), where he was fired for incompetence. He wound up with us and within the year was the highest role.

We have lost countless years of experience due to people leaving over this guy and his cronies. He has no formal education, hasnt been in the plant life long enough to know anything, constantly makes idiotic decisions, etc.

These are the people who wont get out of the way of the people doing the actual work, and make their lives/work measurably worse due to their stupidity, greed, and incompetence.

A tale as old as time, yet for some reason we keep letting it happen.

> They are either greedy or stupid. You have to pick one.

Nah. Some people are definitely both, and it's the "stupid" bit that causes the "greed" thing to be applied blindly therefore achieving the opposite of what they wanted.

ie they make poor decisions and lose money instead

"Penny wise and pound foolish" is a real thing

Greedy but too much short term thinking is stupid.

They are too greedy and stupid to give 7% raise for retaining talent when market price is 20% raise.

Talent has to be hired at market price or why would they join your org?

They're greedy, so they refuse a small pay bump to keep the current people happy.

Their bet is that they won't actually leave if they get rejected on the raise. This is where the stupidity comes in, they actually think people would be loyal and stay despite requesting a tiny raise.

What happens instead is that people are, obviously, annoyed and start looking elsewhere, where they'll get a much larger pay bump anyway. Now the idiots are forced to look for new people at much above what they could've paid to keep the old person, because no one is going to accept a new job below market rate.

So yes, stupid and greedy.

Isn't it also possible that this sequence of decisions is "game theory optimal"?

Maybe, but running your business purely based on game theory is in itself pretty stupid.

You'd have to define "stupid".

Paying more for worse staff is stupid. In some cases the employee is good at their job but has languished for so long they’ve burnt too many bridges to be promoted. The only response for these people is work to retirement or job hop.

Some of it is greedy incompetence but a lot of it is just incentives structures.

It is easier to make the justification in a lot of orgs that you need to go out and hire someone to replace someone who left at "market rate", than that you need to pay someone x% more, based on work that is probably nebulous to whoever you need to make the case to.

I don't think there has ever been a time where this has not been the case outside of very small companies or niche operations. The same sort of incentives are endemic in a business of any scale, because ultimately org structures end up as pyramids and people will intrinsically compete to be at the top

That's no kind of answer! The system made me do it is not an answer! Recall the incentive structures are made by management. So who in hell is in charge here? Management or the paperwork? People or the small 'c' culture of the company? There's no way to double talk or tap dance you're way out: management blew it. 9% is less than the new guy's pay. The old guy is pissed which is why he wrote post. And the rest of us have recorded one more reason "vaunted" American management is stuck on stupid.

(Note I am American and work for American companies. I've had good experiences and terrible in the ol' USA. We have the fundamentals here but damn it too many management people just don't listen to it.)

> Recall the incentive structures are made by management.

I think more incentive structures naturally emerge over time. Managers have other managers who have other managers... etc who have some different set of incentives. There are also shareholders

A lot of time the right smart thing is done anyway, but usually that's luck in having a couple of good smart people in your org chart chain, or just the rising tide of the economy making it easier for generosity to prevail

Management "blew it", but there were probably no negative consequences to them in the case. There are negative consequences for the shareholders when this sort of stuff happens over time in the aggregate... which I believe you get to call the "Principal Agent Problem" if you earn an Economics degree or understand game theory or something...

I hear you ... and in fact you make a good point re: couple of smart guys (or ladies - sometimes the guys are the problem). It's essential to have people with enough self awareness and security and medium to long term thinking to not bow to as I perjoriatively labeled it small 'c' culture (as opposed to the good cap 'C' culture). Culture like political parties are only as good as the fundamentals they hold up long term.

Part of that skill is knowing what can go wrong. And to that end there's no better short read than Ishikawa's tqm the japanese way.

I also agree with your other point: defects are often seen in aggregate when unfortunately the damage is done and cynasysm and infighting are almost unstoppable.

Unions reward seniority over everything else, including skill. I'd much rather have the current environment of being able to leave for something better without starting over again at the bottom of the pile.

> Unions reward seniority over everything else, including skill. I'd much rather have the current environment of being able to leave for something better without starting over again at the bottom of the pile.

Let's see if you still hold that opinion in a few years when you reach your 40s/50s, can't find a job because ageism, and you have to compete with people who work 16h/day surviving on cold pizza and Red Bull.

Already well into that age range and have no trouble finding jobs because I'm productive and continue to upgrade my skills. Those habits are rewarded, as opposed to rewarding you for simply existing longer/having gotten hired earlier.

> Already well into that age range and have no trouble finding jobs because I'm productive and continue to upgrade my skills

> I

"It's not happening to me, so it's not happening" says the programmer type, who isn't carrying 25kg loads up stairs 10+ times a day

I may be a "programmer type" but I do know 25kg isn't a lot of weight to carry around.

The article is specifically about tech workers in Amazon, so I think it's pretty clear from that and the rest of the context that I was discussing unions in relation to tech workers.

I was disputing the claim that you a) it's hard to get a job in your 40s or 50s, and b) if it is hard, it's because of ageism and not something you did (or didn't do). And as others have pointed out, there is certainly a class of employee who is going to be chugging energy drinks and working 16h a day but if you're 40+ you're not competing with them.

I suspect much of the ageism in tech comes from talented/successful people filtering out of the mainstream workforce by 40s/50s. I think a lot of very skilled people end up with the means to focus on other parts of life, or to do whatever they want in tech and not be beholden to a corporate master.

Which isn’t to say that everyone who doesn’t retire by then is bad, just that the ratio is different than 20s and 30s and this probably colors people’s view of the group. Especially with the crazy industry growth we’ve seen over the career of the average 50 year old in tech.

The idea that a lot or even a sizeable percentage of tech workers can "focus on other parts of life" or "do whatever they want" within 15-20 years of working is frankly kind of ridiculous.

The median developer salary in the US is in the neighborhood of $110k/yr and includes $0 in bonuses and $0 in stock. Outside of the San Francisco bubble simply being able to type JavaScript into a computer does not put you on a trajectory to retire decades before your peers in other industries.

Is there significant ageism in other parts of the industry though?

I thought it was typical for, say, a regional bank software department, to be staffed with a mix of ages up to retirement age, not mostly people in their 20s and early 30s.

All of the ageism concern I’ve seen discussed has been about big tech and startups. Are 48 year olds being pushed out of working at mid sized insurance companies?

The median includes the massive number of people under 30 who joined in the last ten years. They likely outnumber everyone over 50 by 10x.

It's not a meaningful reference point for this discussion.

Of course it's a meaningful reference point. It is the primary reference point when discussing how much someone can expect to earn in a given profession. Half the developers in the country are making less than that and it's not because they started programming in 2018, it's because 99% of developers do not get any bonus, do not get any stock, and are lucky to hit $200k total comp in their life without going into management.

If you're going to say something like 90% of the workforce is under 30 you're going to need to cite something because I just absolutely do not believe that.

> have to compete with people who work 16h/day surviving on cold pizza and Red Bull

Even at age 36 that’s no contest. If they’re producing crud twice as long that just means there’s more crud.

I’ll admit that that might be harder to see for management than the fact they spend twice as long in office.

If you had such long career and still compete with these guys, perhaps consider a different career. I'm in my 30s and have no reason at all to worry about this, as my job is no longer about crunching code as quickly as possible - my input is on a much more strategic higher abstraction level that these younger code crunchers don't have the necessary experience for. The VP that manages me would consider 16h workdays and Redbulls a fireable offense at my position.

> Let's see if you still hold that opinion in a few years when you reach your 40s/50s, can't find a job because ageism

Unions forcing companies to pay him higher due to his age makes ageism worse, not better.

Yup. I watched a friend get fired because the union salary was double for people with 20 years of work. Yeah, they were more efficient and knowledgeable, but not 2x. So they replaced him with a younger new grad.

What union contract allows firing without cause?

Pretty much all of them, really. Business concerns are always important because the union doesn't want to kill the business. That would be really bad.

In the case of this example, the manager was given a budget and that meant choosing staff. So she told my friend that he could go the easy way or the hard way. There are plenty of causes that the manager could dream up. It's not hard. If you want a cause, they can find one.

> So she told my friend that he could go the easy way or the hard way. There are plenty of causes that the manager could dream up.

That's avoiding union contracted processes. Union representation can help a worker when management is concocting bullshit issues. The "hard way" might make things more difficult for a worker (closer supervision, tedious training) but it's definitely harder for the manager than if the worker quits.

>Let's see if you still hold that opinion in a few years when you reach your 40s/50s

Do you think none of us are working in our 40s and 50s?

Well, I'm 62, and I still hold that same opinion.

I'm in a union and they don't "reward seniority over everything else". In my experience this is either made up or affects some narrow class of jobs but is used as an excuse to bash all unions. As another response says, unions do what their members want.

Pilots bid for jobs based on seniority and are laid off in reverse order of seniority. Same with police officers. Every union job there is has a concept of seniority number and there's no way to get around that unless you go into management in which case you're no longer in the union (such as smaller police departments where the lieutenant and/or chief are not under union purview).

The screen actors guild hires based on seniority? That’s why Clint Eastwood keeps playing 17 year olds…

There are plenty of unions that aren’t seniority based.

They also don't hire anyone. They are not the ones who do the hiring. Airlines it seems - for a current example - have relinquished some hiring / firing authority to union rules.

I'll listen to arguments for a developers' union that includes no seniority-based provisions but one of the biggest reasons people cite is "ageism."

Seems to me (most) unions and old-style corporations reward seniority because they are captured.

(Roughly speaking) the old-timers stick together because they can and because there are old-timers. And few people get promoted because there is only a very narrow path upward (say, 6-10 lower to a ground-floor supervisor, 6 of these to a higher rank). That necessarily creates a very thin path up. And there is space up only when the previous person is promoted or leaves. There is respect for the skill of a senior employee when it has been proven but respect will only buy you so much. Mostly that skill is put to good effect without massive salary change. There is nominal seniority-based salary increase because that's affordable and easy to manage - and is captured. That is, this salary scale was designed by the previous seniors. Same for promotion order and layoff order.

Seniority then becomes a way to "force" loyalty: If you jump, you may not be one of the top dogs who think they have captured the system. No need for (much) higher salaries if the staff themselves support seniority.

My brother works in a grocery warehouse in CA. Their union seems very seniority focused. He’s worried about being laid off but doesn’t want to look for a new job as he said his seniority clock would start over at the new place.

Don’t you have like scales that automatically increase with age? Of course there’s different levels of scales, but I’m fairly certain there is an age component.

No. I work in tech and pay works the same way it does for everyone else. The union is: https://prospect.org.uk/

One thing to understand is that while the US and Europe both have a thing called "unions", they are different enough beasts that any comparison between the two are essentially meaningless. Even comparing "unions" between European countries is pretty tricky.

Unions reward what their membership wants them to reward.

Are you active in your union? Have you voiced concerns over this policy of theirs?

Democracy does what their people want, yet about half the population are really unhappy after each election. You as an individual can't change things in most cases, you get what the union gives.

Which is why all those unionized hollywood actors DON'T make $80 million for two movies right?

If you are truly valuable, no amount of union anything can prevent you from getting a good deal.

>>Unions reward seniority over everything else, including skill. I'd much rather have the current environment of being able to leave for something better without starting over again at the bottom of the pile.

You are arguing against the very idea of a Union. The whole point of a union is people sticking together for long to tend to each other's interests.

Why should a union reward someone known to be a grifter?

The people with stake in the game (seniority, been together longer) can stick together and continue to drive what they think has worked for them (seniority). I haven't needed to mention skill or effort or initiative one bit here.

While the newcomers have little stake in the game, perhaps don't even vote, vote in dispersed order, etc.

>The whole point of a union is people sticking together for long to tend to each other's interests.

I spent almost a decade doing union labour in Canada (auto industry). The idea that they "stick together" any more than any other group is not my experience.

During one of our negotiations, the company proposed making everyone hired after a certain start date a "temporary employee" (no benefits, lower pay, can be laid off or recalled with 24h notice). This would affect 25% of the employees. In return, the company offered a bigger signing bonus: $2500 vs the usual $1000.

The union leadership recommended voting against the contract; the fact they even brought it to the table is crazy. And the union membership voted it in anyway. Screwed over fellow workers for $1500 a head. It was that easy. Us "temporary workers" were now working beside people making 30% more money, doing the same job but facing more uncertainty. We were gradually laid off over a couple years, all in the name of "corporate viability". I had spent years hearing about "solidarity", but the second an extra paycheque was on the table no one cared about their "union brother".

Joke's on that union, I guess: the plant closed during the 2008 financial crisis and everyone lost their job. Hope that $1500 lasted!

Who is a grifter in this scenario?

I have no interest in rewarding Adam just because he happened to be born 3 years before Bob.

If Adam was there with the Union for 3 more years, then union obviously rewards Adam more than Bob.

The word 'Union' itself means a association of people with a purpose of taking care of each other. You are free to not be a part of such an arrangement. But that doesn't mean that the association is wrong, or its members shouldn't look after each other, or they are wrong to do so.

In fact you are free to do whatever you want, and they are free to what they want. Its just history shows that if you can win big individually(outlier) you are not likely to love unions. But very few people can be outliers so for the bulk of the human race unions work just fine. You tend to do well on the average on the longer run.

Incidentally, this job-hopping is bad for software, because too much institutional knowledge is getting lost, and the job-hoppers don’t get to experience the long-term consequences of their design and implementation decisions.

It can be good for software too. Best practices and new ways of working slowly dissolve through other companies as employees move around. Many of the great technologies invented at the FAANGs eventually get recreated at other companies or in open source because of this movement.

> Best practices and new ways of working slowly dissolve through other companies as employees move around.

Doesn't help against cargo culting and reinventing wheels Just Because We Can And Have The Money.

Just because Google did Kubernetes everyone else followed suit despite there being competitors available (e.g. DCOS, but that one sadly failed because it was utter bananaware). Everyone followed Google for Angular and then went over to Facebook's React, and on the backend side it's Go here and Rust there.

All of those were made to solve problems with their predecessors and I believe that almost every one of them (Angular is debatable) has been a success in moving the needle.

> All of those were made to solve problems with their predecessors

This is not really a question, yes. But... they're sledgehammers and most of the people used them because "the large ones are doing it" - for a loooot of use cases simpler solutions such as plain old jQuery+SCSS / Portainer or Docker Swarm/Compose would have been more than enough.

I have worked with both DCOS and Kubernetes - the latter is way more coordinated and polished, with also better fundamentals underneath.

This results in a situation where a lot of different places have logical, reasonable reasons to pick k8s because it matches reasonably well 80% of their needs, and while everyone's 80% is different, there's enough overlap. This pulls in, like frame dragging, others who might not actually need it, but it's not because some scaling memes.

In practice, a lot of initial intake for k8s was not bananas scaling arguments, but it actually solving problems for people who either had growing complex messes, or considered the spend associated with "good practices" that are often thrown as argument against kubernetes, like running separate cloud instances for every application.

Similarly, everytime I look at Nomad and related stack, it turns out that various bits are missing, and DIY will be harder than with k8s-like stack because it's not properly designed to be extensible.

Good companies will allow time for documentation and testing[0], and great developers will make time for it whether it's officially there or not.

[0] Mediocre unit tests are as good or better than pristine API docs that are slightly out of date.

Having good documentation is great, but there is nevertheless a substantive cost in high turnover even if you have good documentation. Documentation doesn’t replace experience with the product or project, nor is it ever complete.

Looking back at my own career, I do not jump ship every 18 months, and I have been able to accumulate deep knowledge that are broadly applicable, that has enabled a lot of options when it is time to go to a new company.

So in addition to having better continuity of institutional knowledge, there is a benefit to staying on longer (at least for me).

A financial benefit?

Versatility. That gives me options as the technology and business landscape changes.

I've seen former coworkers brag on their CVs about "highlights" for work they did that was nothing short of an unmitigated disaster. But they didn't stay long enough to watch the meltdown.

Middle-upper management also excels at this method of escaping blame/consequences. Move in, talk big and change a bunch of stuff, then on to the next before the tickets start coming in.

The thing that is different is the ridiculously high demand for developers since the 1990s, minus downturns like the dot com bust. Developers can job hop and get perks because software is still in the process of eating the world.

But no boom lasts forever. In other creative fields or games development, where supply far exceeds demand, the situation is much less favorable for workers and that's eventually where the software industry is going to be as well.

I am not so sure that's true. Almost anything that gets created today involves some software, and infrequently some new software, in the pipeline.

Do remember that there is a whole bunch of software that needs to be created for our household appliances, any electronic device (those tiny little SoCs that control a sensor or voltage or whatever), or tools used to make anything else. We are also in the midst of a push to get everything "digitized", including our entire homes, cars, buses, libraries, movie catalogs, music libraries, battery charging, communication infrastructure etc. And everyone keeps a computer or two or three (phone, smartwatch, tablet, fitness trackers) on them at almost all times — all these need software to do something useful.

Basically, I think software developers are going to remain in need as long as we rely on computers. And just like farming has never gotten obsoleted because we continue to need food (though increasing the brute force of the tools we have has allowed us to rely on fewer farmers), we'll continue to need new software (it's still an open question if there are tools like Copilot that will enable us to achieve similar efficiency improvement and rely on fewer software developers, but my guess is no).

>is it the 21st century, something fundamental about the industry, or the fact that software people feel they have more mobility, and thus less loyalty and patience?

Well unions are pretty well known for prioritizing seniority above all else.

It's one of the biggest negatives that come up against them.

> Promotion is primarily seniority-based, and the company rewarded loyalty as much as, err, value creation.

Because truckers don't "create value" the same way that software engineers do. If you had a company of 100 of the "top 0.1%" truck drivers, maybe you could charge a bit more because your insurance would be lower and you'd have fewer accidents and late deliveries. If you had a company with 100 of the "top 0.1%" software engineers there's a good chance you're on your way to having a company worth billions of dollars.

Truckers with solid driving records and good ability get specialized jobs.

If you have the top percentile of drivers chances are you’re making heaps of money moving oversized loads, nuclear materials or running time sensitive contracts.

You trained yourself, you paid for your schooling, you continue to learn on your own dollar for the company. The company doesn't pay for any of that anymore like back in the day, so yes, you must do what you can to keep your career flowing, jobs are just stepping stones, they aren't entitled to loyalty, because they stopped being loyal first.

Every company I've worked at has given me a yearly budget for education, or has just had a rubber stamp policy of paying for classes. I know a few older people whose companies paid for them to take night classes, but I didn't know it was as common as all that.

Consider yourself lucky. I haven’t had that in multiple decades, and no benefits at all for one.

Then California came for my contracting job.

> is it the 21st century, something fundamental about the industry, or the fact that software people feel they have more mobility, and thus less loyalty and patience?

I would rather phrase the question as: why should any employee have loyalty towards a given company? The main goal of companies is to make money (if they could make money without employees, well imagine that). The main goal of an employee is also to make money. People don't usually work for free (there are exceptions, of course, I'm talking here about the vast majority of works and employees). Given these statements, it's natural that:

a) companies get rid of employees that don't make money (unproductive employees). Layoffs happen every single day

b) employees switch to other companies for more money. Obviously, this is easier to do in some industries than others, but the essence is the same in all of them

> Layoffs happen every single day

It didn't use to be that way, which is what OP was trying to get at. Companies used to reward loyalty with bonuses that scaled by tenure, and tried to retain workforces that they spent considerable resources on training, even through lean times.

Yeah the main issue I see now is that the only way to get more money is to be promoted into usually management. That causes everyone wanting to be mangers even though a lot of engineers should not be. However, they refuse to pay a senior engineer who runs circles around everyone more because they are at the top of the pay band. I would be perfectly fine staying at the same job with no promotion if it meant I got yearly raises/refreshers that outpaced inflation. In reality you have to job hop or get promoted to get a meaningful raise.

Things used to be more local, and less cynical. You're of course correct in your assessment, logically speaking. But with a shred of empathy—on both sides—it is also clear that loyalty to your employer is beneficial to them, since they can rely on their workforce even if they face a rough year, and loyalty to your employees is beneficial to them, since they can rely on a stable job, even if live gives them a hard time. Both situations do occur all the time, and are just part of human existence.

So I wonder if this is either the end-stage capitalism everyone is talking about, or the community aspect got thrown under the bus somewhere.

Yes, there's also that side. You're right. When I wrote my comment I was thinking more about the classic tech company/startup. But definitely there are other kind of companies more local (and in the past they were more common).

> company rewarded loyalty as much as, err, value creation

Companies reward loyalty because they can keep you behind the market rates. Hiring a new person costs you market rates. Keeping you around with 2-3% increases every year with occasional promotions of 5% will ensure you stay behind the market.

I think the key is consistency, and it is consistency that is hard for a company to guarantee. If a company does not promote often, like Apple, employees will be content to stay where they are. But once the company starts to promote some people in a short time frame and especially some think the promoted do not deserve the promotion, the employees will start to envy, to demand a speedy promotion, and the company will slowly, sometimes even quickly, turn into a promotion-oriented culture. Case in point, working in Google was such a prestige 15 years ago, and few people would even think about reaching E6. What about now?

That loyalty makes zero sense when one will be downsized in 0.1 seconds if it would make the CEO 1% richer and staying in the same role means getting paid less than market rate including new folks in the same org.

As a uniom fellow he might well be overpaid if he at 30 years in makes a lot more than the 10 year guys while doing the same exact job as them.

Basically people like your dad were getting tithed by management for loyalty while wondering why you don't tithe your company for security in a market where the most important goods require 2-5x as much.

Many blue collar workers in the 70s could afford a house, a car and a wife who didn't work to take care of the home and kids.

> 35 year career

look what they took from us. I've been working for 24 years and I expect I'll be working 25-30 years more.

Just prestige and start your career over. Not everyone wants what you want.

The software engineering paradigms used within the company create brittle rube goldberg machines of events flowing everywhere in the company.

Thank you for saying this. I have interacted with various seller-facing Web software platforms (KDP, Ads, Advantage, Seller Central, Transparency, Brand Registry) for years. Hurriedly designed features, broken processes, false-positives, and kluged-together garbage are the norm. Automated support scripts, overwhelmed human CSRs, and the siloed nature of the company compound the problems.

Here's one small example: Amazon forced many sellers in the US to start selling in Brazil. It was automated. People's accounts were enabled for Brazil by default. It was poorly communicated, including no easy way to opt out (redirecting links on the help page). Many sellers in a panic started requesting support to shut down their Brazil accounts, which led to entire global accounts being shut down.

https://sellercentral.amazon.com/seller-forums/discussions/t...

> The software engineering paradigms used within the company create brittle rube goldberg machines of events flowing everywhere in the company. Almost all of them are on maintenance mode, where the oncall burns out the engineers and prevents them from creating new products. There is no knowledge sharing between team members. Legacy team members guard their technical platform knowledge to solidify their place on the team.

I have never worked at Amazon. But I did work at a company that decided to implement microservices a la Amazon, even going so far as sharing Bezos' famous 2 pizza team memo. This effect essentially happened over night. As people spun up more and more microservices, things got more and more siloed, cross team collaboration was significantly more difficult, and things became an increasingly more complicated rube goldberg machine that just destroyed people with on call schedules.

Software engineers having "on call" schedules at all is crazy to me. You shouldn't be writing code at 3am to fix a bug after working all day just to turn around and work the next day as well.

It works well of the company empowers engineers to write software that doesn't break all the time.

At HBO Max every incident had a full writeup and then real solutions were put in place to make the service more stable.

My team had around 3 incidents in 2 years.

If the cultural expectation is that the on call buzzer will never go off, and that it going off is a Bad Thing, then on call itself isn't a problem.

Or as I was fond of saying "my number one design criteria (for software) is that everyone gets to sleep through the night."

The customers win (stable service) and the engineers win (sleep).

What was the time to implementation for the real solutions?

Was there any cost considerations associated with prioritizing the implementation or even limiting the scope of the solution?

The tooling there was amazing so a barebones services could get deployed into prod in a couple days if needs be.

That typically didn't happen because engineering reviews had to occur first.

A single command created a new repo, setup ingress/egress configs in AWS, and setup all the boilerplate to handle secrets management, environment configs, and the like.

All that was left to do was business logic.

It sounds like if you have production issues, there is a priority to put out an actual fix immediately.

It depends on the severity of the issue.

If the issue impacts tens of millions of customers, then yes, get it fixed right now. Extended outages can be front page news. Too many in a row and people leave the service.

Ideally monitoring catches outages when they first get started and run books have steps to quickly restore service even if a full fix cannot be put into place immediately.

My experience being "on call" as an engineer has mostly not been that you need to write code at 3am. It usually comes down to restarting a machine, deploying a new machine or copy, or informing the rest of the company that some 3rd party API that you rely on is currently down.

But that's not engineering work, that's technician or operator work. The engineer comes in later to discuss what went wrong and how to prevent it next time.

Speaking as a technician whose seen 3 AM at work many a time.

I think the devops-inspired idea of engineers owning what they deploy has become fairly popular, for better or for worse.

For my own part, this wasn't a huge team. We had the knowledge of if the issue was application/software based but would pass back to ops if it was hardware/OS related.

One possible bonus, being on call operating your own software also gives you a solid incentive to not wake yourself up in the morning by writing bad code, and fixing those issues that do arise quickly.

> being on call operating your own software also gives you a solid incentive to not wake yourself up in the morning by writing bad code

Unfortunately, my software interacts over network with software written by other people; if something goes wrong at 3 AM the users don't know which part caused the problem, so they wake up a random person.

if your pagerduty/equivalent is doing that, something has gone very wrong at your company

That’s an observability problem, not an on-call problem.

When I worked at Samsung Austin Semiconductor, it was absolutely the norm to call or text engineers after-hours to weigh in on machine irregularities.

As others alluded to, there's no reason for an engineer making $200k/yr+ to do that. You can document how to recover from those error states and pay someone 25% to handle that.

Seriously! Working weekends in retail when I was young, one of the hallmarks of a "real, professional job" was not having to work nights/weekends when your routine schedule is during the day. It was a major motivator to get through school and get skilled.

Now I see young engineers from top-tier school working "on call" without complaint. I've found ways to avoid such roles, but it always seemed ridiculous and completely unnecessary in a world where there are software engineers around the globe that could easily work full time support positions.

I found it to be rather the opposite; when I was off from a wage-slave job, I was actually off. If the boss called, you could just ignore it and say you missed it because you were studying or sleeping or with friends or whatever and they couldn't really say anything because they knew they didn't pay you enough to care.

When you're making real money, they own your ass.

Indeed, those who are exempt or whatever with salaries... consider it purchase/lease with at-will employment in most states.

Both salary and hourly gigs have income hanging by a thread with plenty of work, yet only one can get overtime.

Be given responsibility/salary for something (aka hired) by a particularly needy manager/org and be 'undependable'. Read: not at their call. See how it turns out.

The worst/eventual outcome: bye-bye money. Hopefully one has a more reasonable environment. Workers have little on their side.

As someone who does SRE (not AWS, elsewhere)... I would absolutely prefer pay as an hourly rate over salary. I don't like putting in more hours/making less money because Developer Kelly had a bad launch... but I have to, The 9s (and bills) Must Flow.

Fortunately, my current place takes this into account. I don't actually need bonuses or structure change... but the larger trends remain. The employer is buying you, salary opens the time box.

It is a self fulfilling prophecy. Unrealistic schedules results in crappy code which results in pagers/alert going off at all hours which results in unrealistic schedules. Agile's answer is that we reduce the scope of things delivered. You might as well spit into the wind. Deadlines are set by the business, no matter what the Agile evangelist said.

What would be more sane? Allowing things to remain broken at night? Having someone who is not a software engineer fix it?

Either pay another full time night shift or yes, accept that things are broken for the time.

If your product is really so important that it can't be down, hire more engineers and pass the markup to your customers.

I'm glad I live in the country where you have to have 11 hours between end of work and start of work(except for special cases afaik).

There are legitimate reasons to pull in someone after hours, but it really has to be catastrophic. I'd 100% want to be called in if I deployed something knocking out 911 service for a whole state and I was the only one with the knowledge to actually fix it in a timely manner. However, most problems are not like that and are either able to be delayed until an actual business day or can be solved by someone else.

Let's be real, we're talking about line of business apps and ecommerce stores making $5,000/day total revenue. "Critical infrastructure" has an entirely different failure model.

If the product that breaks in the night is SO important for the company, well, why is not the company paying for dedicated people (not the engineers who create the product) to take care of it when it's broken? As said above, while on-call you don't write code, you just turn off feature flags, reboot machines, etc.

If the company cannot afford that, then the product is not that important and can remain broken until the morning.

Even 24h fast food places hire 3 people (each working 8h)!

If a 7-Eleven is open 24 hours a day, they usually hire three 8-hour shifts (roughly speaking).

If you want things to not break, have redundancy in hardware and failover modes that let you function in reduced capacity.

Manual fixes should never be done in a hurry, and if your system is that fragile, I really wonder about the competency of your senior employees and leadership.

> Software engineers having "on call" schedules at all is crazy to me. You shouldn't be writing code at 3am to fix a bug after working all day just to turn around and work the next day as well.

Oncall is only crazy to anyone who also believes it's totally acceptable to have whole services down for hours throughout the night.

To those who understand what it takes to have anything available 24/7, you understand damn well that you need someone to jump on a laptop as soon as an alarm bell rings.

Well, that someone better be someone else than me, because I'm not going to do unpaid night shifts. If you want something running 24/7, it's surely important enough to warrant hiring someone else to take care of it while I'm asleep, no?

Keep your fancy valley salary (with the ridiculous rent prices attached), and I'll keep my European workers right's protection—including undisturbed sleep after my 8 hours workday.

It certainly shouldn't be unpaid. When I've been on call we got half a week's salary for the week (in EU).

Mostly things went smoothly so that's a pretty nice bonus.

Yeah, it's different in the EU. In the US, it's often expected from engineers to be on unpaid oncall—that is, these companies usually phrase being oncall as part of your ordinary duties, without additional compensation. And even if it's compensated, sometimes you cannot opt out of this without seriously harming your career.

Something ridiculous like that is luckily impossible in (most?) EU countries.

What happens when you are drunk or at a concert? What happens when you do not hear the phone (or if it is off)?

Is the effect unformal harm?

In the EU when you are on call this is a contractual thing.

Is it an US thing that on-call shifts are unpaid?

That is not common in Europe. Generous compensation and additional time off is quite typical for engineers handling on-call burdens.

Yes it is.

At one company, I was technically on call 24 hours a day 7 days a week for over ten years. Did I get called that often? No. Did I get called at the worst possible moments? Yes.

That's not a problem with the concept of being oncall. That's an entirely different problem that's not technical nor operational not industry-specific.

Isn’t the fact that you receive calls seldomly, but at the worst possible moment literally the core problem of being oncall?

And it’s certainly industry-specific. Some doctors have this, firefighters—and software engineers. Contrary to the first two, they usually don’t save lives, but revenue though.

Such a weird take.

There is a cost to having on-call. Whether it's in the extra hours you are paying your engineers or other technicians, or sleep deprivation, dwindling motivation and performance, the cost is always there.

In a business, cost is always balanced with the return on that investment.

So it trivially follows that on-call only makes sense where the return is bigger than the investment. If you are having your $100/h engineers become $20/h engineers during the day because of the on-call rotation, and you lose $200 of sales over night when things are down (even your customers are asleep) — you are actually investing that $80/h difference for 8 hours ($640) to recover $200, for a net loss of $440.

Yes, there are cases where it's fully acceptable to simply have your service down for the night. Eg. imagine a service that provides the amount of energy sun is providing for a location (to combine it with solar farm production): is it really that bad if that's down at 2am? Sure, it might be nice to get it back up before the sun is up, but this is just a trivial example where an uptime of ~70% (fluctuates) is perfectly acceptable.

> There is a cost to having on-call. Whether it's in the extra hours you are paying your engineers or other technicians, or sleep deprivation, dwindling motivation and performance, the cost is always there.

I don't understand your take. Every single time I had a job with an oncall rotation, that oncall was paid. I was paid a bonus for being oncall, I was paid a bonus if during oncalls a pager fired outside of office hours, I was paid a bonus if I was pulled into an incident response outside of my oncall rotation. There was always a cost, and we were paid for it. Being oncall represented loosely a pay bump of around 15%.

If that's not your case then I'm sorry but your problem is not the oncall rotation.

Getting paid for on call, especially this tri-level multiple bonus structure, is incredibly uncommon.

That should make my point more obvious: why would a business pay you 15% more if they are losing minor or no money or customers if services are down until someone comes back for their regular work day?

If what we’re talking about is a website/app/SaaS/etc, and if it needs to be up 24/7, then that almost certainly means that it’s being used globally, or at least across several timezones.

So, hire a team in another time zone.

This is a problem of management not prioritizing the health and wellness of their employees, simple as that.

It's absolutely acceptable to have your website go down for some reason overnight. Fix it in the morning.

Even if your app is critical infrastructure (it isn't, and 99% of you shaking your head and saying it is are objectively incorrect), you don't need a software engineer to fix it. You need an SRE. That's completely different.

> But I did work at a company that decided to implement microservices a la Amazon, even going so far as sharing Bezos' famous 2 pizza team memo. This effect essentially happened over night. As people spun up more and more microservices, things got more and more siloed, cross team collaboration was significantly more difficult, and things became an increasingly more complicated rube goldberg machine that just destroyed people with on call schedules.

Micro services solve a problem for companies that have already reached a certain scale were cross team communications have become unfeasible and now communicating by well defined API contract is a better choice.

Also ideally micro services should reduce the blast radius of outages. If an outage is not easily traced to what team is root causing it, then proper monitoring is not in place.

Sure I've had times where on call went off and it wasn't my team's fault but it was no more than 10 minutes to determine that and reroute the call and then to back to sleep.

The other fact is that when designing a new microservice it should be done in conjunction with the primary consumers of that service just like designing any other part of software. Stakeholders need to be brought in and consulted.

The advantage of microservices is that new code is only going to impact direct consumer s and downstream services.

It also allows you to upgrade a service in place or completely rewrite it so long as you adhere to the original contract.

I've seen impressive rollouts of security updates across a huge code base that was only possible because of a microservice-based design.

I've seen giant monoliths fall apart as multi-year long efforts are undertaken just to update the build system.

The hard part of a microservices is they require discipline in an organization and basically assigning an engineer for 1 to 2 weeks to write run books and add monitoring.

Of course, those runbooks should be written no matter what paradigm someone uses!

At Amazon's scale is there any alternative to a service oriented architecture?

SOAs serve the purpose to more clearly delineate responsibilities: any appearance of tight coupling is made relatively obvious.

Nothing stops someone from simply enforcing the same division in a single large code base. Your API contract can be your public API in whatever programming language, and this would allow you to work with the same assumptions from the SOA.

It would only be easier to break out of the recommended way of doing things, but you can provide simple tooling that does static analysis to prevent that (I remember using Zope3 security configuration to achieve exactly that with Python code in ~2006).

If you are concerned about a performance from such a large monolith, you could be using a functional language (or at least the pure functional paradigm) that allows easier infinite horizontal scaling.

> Nothing stops someone from simply enforcing the same division in a single large code base.

I'd say nothing except human nature.

When I said "enforcing", I really meant with static analysis tools.

> The big problem is legacy employees run every part of the company

The article comes to the exact opposite conclusion.

The article repeatedly cites problems associated with culture breakdown, and tenured employees are a key source of continuity for culture. The article goes to some lengths to underline that the culture is a positive and key differentiator for Amazon.

The more interesting point is that the cultural element looks like it is highly durable and effective, but has broken down as Bezos (the most tenured employee!) has pulled back. So it's not as durable as everyone thought.

I don't think those are mutually exclusive conclusions.

Tenured employees help maintain culture, but they don't guarantee that it stays static.

Culture developed by the employees who spent their first 5 years growing with the company is not necessarily the same culture developed by those same employees who have stayed in the same role for the last 5 years with a vanishing potential for growth.

If the culture you want to maintain is "first 10 minutes of every meeting is everyone reading the agenda" or "thorough security review for all changes before go live" then you want slow turnover so new hires can be trained and indoctrinated.

On the other hand if the culture you want to maintain is "constantly be trying out new technologies, new ways of working and fresh ideas, at the cutting edge nothing is set in stone" you might benefit from new employees who know firsthand how other companies solved whatever problem you're facing.

Absolutely. A company that goes through an extended period of growth and success can retain (cargo-cult style) all the approaches from that era even while the conditions change. That ossification is obviously not great unless the culture is especially adaptable.

And the conditions for most big tech companies have changed radically over the last few years. So the subtle devaluation or reinterpretation of culture is kind of inevitable when forever-growth turns into a zero-sum game. As you say, it might even be the veterans who are doing it.

Culture is not singular, but the summation of many different desires and behaviors. The behaviors of those who climb and then defend a position is likely different from the people who change positions every few years. As time progresses, more of the defenders stay, and more of the changers leave. This would shift the culture over time, not because it’s made up of different people, but because only a subset remain.

> Almost any manager of managers has been at Amazon a long time, in that same job for a long time. There is no upward mobility at the company, unless you have been in some org 5+ years.

This is not specific to Amazon, it's endemic to the whole industry (and probably outside of the tech industry, for all I know): Very senior leaders and engineers are sometimes where they are because they've clung to their role for a long time, and they don't leave or die at a rate sufficient to frequently bubble up new senior talent through internal promotion. These people may also deserve their position because they are brilliant, too... or they may not. I've worked in a place with technical leaders where nobody knew what they did--their job was apparently "Being Employee Number 3" and nothing more.

I was at an organization that went from doing mostly smart things to doing mostly stupid things.

One of the smart things they managed to do was recognize that they were top-heavy with senior+ talent who were fixed in their thinking & behavior and aggressively began recruiting new graduates and reaching out to CS/Engineering/Science seniors. They also added some pretty impressive mentoring programs to bring the new people up to speed quickly.

Then they switched to doing stupid shit that mostly undermined all that and the newer people started leaving. But it was pretty good for a couple of years.

Ok so let's say for the sake of argument it's a Smart Thing to focus on hiring a bunch of 22 year old fresh graduates. Is that necessarily going to change anything at the leadership level? The GP is talking about managers of managers, I don't know where you've worked in the past but in my experience Directors and VPs are not taking a lot of advice or insight from the entry level new hires.

Not directly, no. The point in this case was understanding that the engineering culture had stagnated and needed to be refreshed. The people ordering the change in hiring clearly understood that it would take a long time for any newly introduced behaviors to percolate through the company. But without explicitly addressing the situation, things would just have gotten worse over time.

The reality was that in a fairly short time, focusing hiring on new grads actually had the intended effect of improving the way we did things. However, the company started becoming more unpleasant to work for soon after that. And a lot of those new hires, now with a couple years' experience, left. As did I, so I can't say how permanent the changes were.

> I've worked in a place with technical leaders where nobody knew what they did--their job was apparently "Being Employee Number 3" and nothing more.

Why does there need to be anything more? They probably know how the systems work and are ready to step in if necessary.

Also, they busted their ass before other people were at the company, enabling the company to grow and create the jobs that all the other people are now in.

To me it's not anywhere near as egregious as the founder who worked for a couple of years, then outsourced all leadership to a CEO for hire, and now does nothing, but still maintains 50%+ ownership of a company.

>This is not specific to Amazon, it's endemic to the whole industry (and probably outside of the tech industry, for all I know):

Seems like this is also the problem with Congress.

Yeah, but let's not forget: In the same position and context in life, statistically speaking, you would/will do the same.

> Almost any manager of managers has been at Amazon a long time, in that same job for a long time. There is no upward mobility at the company, unless you have been in some org 5+ years.

That's not true, at least not in AWS. In fact, I'd argue that a big problem in AWS is that promotion is too fast too soon. Getting to L6 used to be a big deal, and many people were content with being on L5. But now everyone expects that L5 gets promoted to L6 in 2 years or even less. And L6 to L7 in 3 years or less. What many directors do is that they place their favorite L6 to lead a project that launches a new product. If the launch is successful, the L6 will most likely be promoted. We even joked that L8 is the new L7. And of course, this creates so much anxiety to the point that more than 50% of the 1-on-1s I'm aware of involved some discussion about "moving to the next level".

> brittle rube goldberg machines

This is a nice metaphor that should become widely known and discussed!

It can be applied to general management and administration was well as software engineering. Organizational procedures tend towards complexity over time: everyone is involved in 'improving' systems by creating new procedures, and soon enough a new person is needed to deal with or manage the increasingly complex procedures. This is a well known micro economic phenomenon. It's hard to simplify and streamline and so being aware of the tendency towards brittle rube goldberg machines is a useful concept indeed, should become Amazon's 17th management principle.

> There is no knowledge sharing between team members. Legacy team members guard their technical platform knowledge to solidify their place on the team.

OH MY GOD!

Thank you for saying this, this is __exactly__ what I've been experiencing. I've seen a bunch of people pursuing their personal (promo) agenda ruthlessly. Delivering on their own projects and initiatives and nothing else.

Hire and develop the best my hairy ass!

I guess don't hate the player hate the game

> There is no upward mobility at the company, unless you have been in some org 5+ years.

I'm sorry if I sound harsh, but if that's your take them either you haven't been paying attention or you're not in the company for that long. The whole org was designed so that all employees are ephemeral. Managers are encouraged to promote employees up and out. At each career level you're evaluated against your direct colleagues in the same level, and those lagging behind are bound for PIPs.

Check out the old fart tool. The average tenure is below 3 years. Do you it screams upward mobility?

Those are just churn and burn lower level employees. I am talking about at the L7+ level. L6 and below is almost a completely different company.

I have worked on major tier 1 services in every major part of the company: Retail, Alexa, and AWS. I know many of the people running the important orgs

The amazon people I know are purposely lounging at L6 because they claim L7 responsibility/compensation ratios for their dept are completely out of wack.

> (...) they claim L7 responsibility/compensation ratios for their dept are completely out of wack.

I have to call bullshit on this take. Last time I checked, less than 1% ever make it to L6, let alone L7. because it is extremely competitive.

It takes a very special person with all the stars aligned to even be considered for a L7 position. We're talking about Principal Engineer/Senior Manager.

Claiming they are not promoted to L7 because they don't want to screams of Aesop's the fox and the grapes.

First of all, they were not making this comment in order to justify why they didn't get promoted. They were describing levels in amazon to people that were clueless about it.

You also seem to be inadverdently supporting their point. If it takes "a special person" to be an L7, doesn't it also require "effort" to become that person? The person is simply saying that they are not even making the effort they think is necessary to be considered for an L7 since they think there is too little upside to justify that effort. To support their point they mentioned that another friend we hardly see anymore became an L7, their intense schedule and workload being the reason we don't see them anymore. It seemed pretty believable in that context.

> First of all, they were not making this comment in order to justify why they didn't get promoted.

You claimed they were purposely lounging at L6. I called bullshit on that, and explicitly mentioned Aesop's "The fox and the grapes". It's unbelievable to claim that they are not L7 because they don't want it. You need to outrun everyone in the whole company, and even then you are limited by circumstances and track record.

An average tenure of ~3 or less years is actually kind of what Id expect if upper management has calcified.

Maybe that's just what happens to companies that have been around a long time. They go on maintenance mode. You don't see coke or visa innovating much either.

It seems like optimizing for innovation makes sense with new upstart products and optimizing for maintenance makes more sense with established products.

You started with if Amazon wants change and I think that's exactly what a product with a position of relative market dominance doesn't want

Smart companies (I have no idea if Amazon is "smart" or not) are always innovating. However, most of the innovation will be internal. I used to work for a large corp that owned 80% of its main market and we were always encouraged to seek new ways of doing things and to file as many patents as possible. They were smart enough to realize that they got where they were by being innovative and that no matter how much of the market they controlled, they couldn't stay there by standing still.

That would make sense, except Amazon stock's PE is currently around 50, which means that it's priced like a highly innovative company with large growth potential. So, if management doesn't want a massive stock price drop (and they obviously don't), they have to structure the company as if it's innovative and high-growth, to sell a believable story to the stock market.

> You don't see coke or visa innovating much either.

Both exist in industries that aren’t innovating much. Amazon doesn’t enjoy such a luxury.

OP's description isn't exactly "optimizing for maintenance" either!

> There is no upward mobility at the company, unless you have been in some org 5+ years

That... seems about right? I wouldn't expect to get to L7+ in much less than that.

Hard disagree.

Firstly:

>AWS hasnt released an innovative product in a really long time

They never released anything innovative, ever. Amazon creates value through optimization. The original AWS came about because they had extra servers lying around that they needed for holiday traffic, and they decided to rent those out.

The problem with Amazon is that its too bloated, which goes against their optimization bread and butter.

The cycle goes like this. First you have an entry level engineer, that hasn't been taught really how to solve problems algorithmically - instead its basically just education in the form of memorization about problems and how to solve them, without anything more fundamental.

Then, companies like Amazon need engineers to actually build the products, so they are forced to tailor the interview process to them, thus the prevalence of leetcode style questions, because they know that the engineers are studying those for other companies.

So the candiates get selected are basically those that have shown that they can memorize processes better than others. When these candidates work, they do the exact same thing internal to the company, focus on doing the existing processes in hopes to get promoted.

As a result you get lots of bloat both time and technology spaces. So no, promoting them into decision making positions is not the right thing to do.

The best thing Amazon can actually do long term is to focus on more automation, and reduce headcount. You want fewer, more talented engineers who are able to solve problems autonomously without barriers in the way, who don't need handholding or don't need to handhold others.

AWS itself was one of the biggest innovations in technology, of all time. Cloudformation, IAM policies, and creating these services as independent composable blocks was revolutionary. S3 was completely new, and is still a dominant force in cloud computing. But my point was that all of the best ideas in AWS came long ago.

There are newer companies, like temporal.io, whose platform capabilities make AWS look outdated at this point.

I think your understanding of leetcode is not correct. It filters for:

Can learn computer science concepts

Is willing to sit in front of a screen and learn said concepts for a while

Is willing to follow directions and recipes

Is willing to follow the rules

Has some minumum bar of intelligence

Unfortunately, these interviews do work. Some percentage of people that pass the interview are actually very good. The amazon firing process will weed out the rest.

Meta has released the best open source software in the last 20 years, and their hiring process is all leetcode memorization

I guess we disagree on what innovation is. I define innovation as technology that either no one has, or capability to do things for a lower material cost.

For example, lets say I have a server rack at home. There is nothing in AWS that exists that I can't replicate on my server rack at home. The only difference is that time required to implement all of it on my server rack vs in AWS, which is optimization.

If AWS comes out with some closed source ML model that is good at doing something, thats innovation - i can't replicated that at home without knowing specifics. Likewise, if they have super cheap EC2s with RISC or dedicated ML hardware that are cheaper per hour to run than anything I can build at home with power costs, thats also innovation.

The interviews aren't fully useless, I agree that leetcode provides valuable insight. However IMO it should be the bare minimum.

> The original AWS came about because they had extra servers lying around that they needed for holiday traffic, and they decided to rent those out.

This isn't true, it's a bit of marketing nonsense.

There was a paper proposing to create the service, it got workshopped quite a bit, and got funded. The servers used weren't extras, it was dedicated capacity from day one.

At the time, Amazon was not using the same hardware (or anything) as AWS. Even today, there are still critical Amazon services running on the older corp fabric, because the Native AWS migration isn't 100% complete.

The other key parts of AWS were also innovation, and not mere optimization.

You have demonstrated that you have a near complete lack of understanding of how AWS innovates. Your ignorance is thorough, and that gives you confidence; but it's unwarranted. You're just another Dunning-Kruger twit.

The actual AWS product yes, but the initial idea came from what I said. Perhaps I wasn't clear - its not like they started renting out servers right away, but saw the potential. As such SQS was the first service that was created.

The corp fabric does run on AWS. Its just the legacy tools that are used to manage the virtual machines instead of them being liked to the AWS account. The hardware is in the same data centers, and legacy tools wrap AWS stuff under the hood.

But yeah flagging for unnecessary personal insults.

what's scary about Amazon is that - despite all this madness - it is still growing and better run than most other tech companies

> it is still growing and better run

what's the evidence that it's better run? continued growth could just as likely (or more likely) be a result of its dominant position

It's subjective, but many (most?) people who worked at Amazon agree that it's an extremely efficient and well-run business comparatively. I've worked at two other companies of similar size and they both were laughably incompetently run in comparison to AWS. They were way better to work for as an individual employee, but certainly bad if you cared about the business operations as a whole (I didn't)

How is it better run if most ex-employees and current think it's a place where to succeed you must hoard knowledge and play a crazy political game where your goal is to always have someone on the team ready to chop. If you don't know who that is on your team it's probably you.

Your pay is equal to three card monty. You are promised something big when you get hired but it gets spread over 4 years and most of it only vests at the end. Then you find out most employees are pushed out within 2 years and you realize only 10-20% will ever survive to get the amount you expected when you joined. If you make it this far they do it again for more money. The odds of you staying are even less...

While this is happening they give you 8 hours of meetings and expect 8 hours of development. You end up working 16 hours a day. You end up so tired and confused you keep going until you burn out. At that point they ppe you.

The best way to survive at Amazon is find dirt on your boss. Hire a pi. Coast and use that information to keep you employed. Rinse and repeat for new bosses.

It is not run well. Taking these practices and putting them on a regular business wouldn't work. Amazon (Google, Apple, Facebook..) will keep increasing in value regardless of management style (each one of those companies have different culture) because of factors outside of this. They could require everyone to be left handed.. or speak French or be able to ski and they would still succeed.

The compensation blew my mind when I worked there.

At the end of my first year I got "exceeds" and a large comp bump (almost all of it from RSUs).

Those RSUs would start vesting one year after this "exceeds" conversation took place--April 1st (start of Q1 a year later).

So I'd get a (small) vest ~13 months later. Considering the average tenure (typically 18 months from what I heard from a higher-up in my org), the deck is stacked against actually making the big comp numbers. You work very hard to make more if you can survive another year after you worked hard. Odds are good you won't.

> average tenure (typically 18 months

how do you get anything done with such a huge turnover? Maybe it's because we're small but it would be impossible for us to accomplish hardly anything useful at that rate of training replacements

Is this from first-hand experience/knowledge?

> way better to work for as an individual employee, but certainly bad if you cared about the business operations as a whole

in other words, "better run" != "employees are treated better"

I got the impression that AWS and retail were two different beasts.

All that matters is market position.

Yes

>>Legacy team members guard their technical platform knowledge to solidify their place on the team.

This thing comes up in a lot of companies, but really the issue is your 2-year term job tourists/hoppers, and there are quite a few of you going around. Its tiring after a while to keep teaching and training every new member who joins your team. This look even more pointless when you look at the fact that they are likely to leave in a few months anyway.

You mostly have your own personal knowledge management system, where all the tribal knowledge goes. Giving people a run of such a large corpus of knowledge/information, especially when they are not like to work with your for more than a few months is pointless.

Eventually you only get as much information as much as you have commitment.

>>If Amazon wants to change they need to remove a significant amount of tenured employees, and actually promote young engineers into decision making positions.

Why would you promote short term people, who are likely to make bad decisions without stake only to fire people who have demonstrated serving in the long term interests of the company?

>>AWS hasnt released an innovative product in a really long time

The whole point of AWS is scale and stability, not newer products.

Amazon seems to have been run in non-ZIRP mode during ZIRP. This may save their asses when ZIRP is firmly over.

You say there isn't any innovation: May be you don't need that in a utility company that moves boxes and bits? You need reliable utility that customers expect and love.

In an alternate ad-supported company where the primary product (ads) is not the one people are most familiar with, you could play all these games of 'upward mobility', 'promotion' etc because money runs freely and your job is detached from the reality of product, customers or even the larger company goals (assuming, you are not directly working on ads).

However, in a company that is truly customer focused (and obsessed - to over use a Bezos word) as is Amazon: all of what you said seems like a good thing?

I do take issue with how they treat the low-level employees with ruthless nearly-dictatorial oversight - Amazon burn out is a real thing. OTOH, having talked to Amazon engineers/PMs who quit due to burnout still have some sort of Stockholm syndrome where they have fond attachment to their 'burn out' time and how boring/unfulfilling their new job at X/Y/Z is! It's Complicated.

Known feeling

   There is no upward mobility at the company, unless you have been in some org 5+ years.

I think that's true of most large organizations.

> I'm a long time Amazonian.

> If Amazon wants to change they need to remove a significant amount of tenured employees, ...

Doesn't this include yourself as well then?

Ah, the 'ol "workforce greening" where we get rid of all the expensive old farts and replace them with younger/smarter/cheaper devs right out of school.

It's brilliant and it always works. There are no downsides whatsoever.

What if it includes them? Doesn't seem like that would change anything.

they could also be stuck under the manager of managers with no upward mobility.

Significant amount != all

And which ones do we keep? Which senior engineers are just coasting and which are among the few who are the final reservoir of institutional knowledge holding everything together?

Does anyone know? Meh, let's just hand the task to HR and hope it all works out.

Ah, so "everyone but me" type deal.

I've always liked the SRE approach, where you can alleviate software engineers to build new features without burdening the team with 24/7 on call duties.

SRE/Ops person here. SRE only works if SRE can look at software and go "Nope, it's not ready for production and I don't care how wanted this feature is." Almost no company is mature enough to allow that behavior.

Same. SRE is generally code for “we need cloud sysadmins but that’s not fashionable anymore.”

You just described every Fortune 500 company.

maybe. i don't know. AWS main services is like an AK-47, it just works, with minimal maintenance. If they started "innovating" like Google with a million different products that get regularly culled, I wouldn't be a faithful user of almost two decades by now.

> Amazon wants to change they need to remove a significant amount of tenured employees, and actually promote young engineers into decision making positions

Hopefully, avoid doing a Boeing in the process.

>The engineers themselves are not students of computer science, but just crunch out tickets.

What does this even mean?

Does it mean they do not have proper CS background? If it means what I think it means, how is that even possible? Amazon recruits at top tier schools, not po-dunk university. Do that have a large % of their Engineering staff from bootcamps?

Being a student not in a narrow sense of being enrolled in a school, but being someone who studies and learns new things by doing rather than simply doing the given task in a conventional/quick/easy way that may not resolve the underlying issues that caused the problems in the first place.

Reply on news.ycombinator.com