Amazon Web Services – Four Years and Out
adventuresinoss.com
adventuresinoss.com
This was for a question about how billing works.
It went like this;
1. Case created.
2. Unassigned for seven days.
3. Open real-time chat, talk for 25 or so minutes where I guide a first-line Indian chap who plainly doesn't know about the subject in hand and who is as we talk reading the AWS docs I've already read. At the end, just as I couldn't find an answer, he couldn't - which is good, he didn't try to give me the wrong answer - he escalates. That's fine - a lot of questions are simple and even silly, and first line support is there to handle them - but they could have done all this without me, if they'd opened the ticket themselves rather than me having to chase.
4. Eleven days later, comes back with exactly the wrong answer. In the meantime, I had figured out the correct answer, and reply, explaining it to him.
5. Next day, I get a wall of plainly AI generated text telling me my answer is correct.
It seems to me a key issue here relating to AI generated text, is a misunderstanding on the part of AWS that I as a consumer will value that answer exactly (or indeed, even remotely) as I would value the answer from a human.
I do not. I almost ignore AI generated text, as I think it as unvalidated response.
Even Stripe - once legendary for the quality of its support - has apparently given up now. I had to deal with it recently over a case where the merchant was seeing an unexpected change in the way it was collecting payments and the AI bot was worse than useless - it actively suggested incorrect explanations and resulted in several days of trying to change the wrong things while the problem persisted.
For my own businesses we give this issue a heavy weight when choosing which services to use. We have even seriously considered moving existing integrations to different services over this one issue recently. If we're integrating with a service then we want to know there's a real person who can actually help if we have questions or anything goes wrong. Failing to provide that because it's cheaper to push everyone through the AI bot is a statement of intent about how much you value your customers.
Instead, you (company / support agent) have decided that I should instead have a conversation with an LLM through a worse, more opaque harness to the detriment of all of us.
But even as an AI believer and thinking it's great for this sort of thing, I also think if it's good enough for this sort of work, it should be good enough to identify when it's not capable of providing value, too.
Meh. It's no big change to before, where you had first-level support search for something vaguely related and just dumping a template to the user.
The thing is... it actually works better than I'd like, because in a lot of cases it turns out that you (as the user) forgot to follow one tiny step in the documentation.
What I'd actually like to see as the user is the AI actually going over my AWS account, looking up the resources and their state on its own, and figuring out what exactly I missed, but (un?)fortunately that cannot be easily done for IAM permission reasons...
I disagree, personally. The way they were employed in GP's example, yeah, that's annoying and hostile.
But almost every time I go to some documentation page for a commercial product now and it doesn't have that, I find it pretty annoying (I don't expect an open source library to wire this up and pay for it, ofc).
It means I have to convey these docs to my own LLM, I will waste cycles and tokens fetching them, etc.
I've been at places where the AWS annual spend was a real lot of money, let's say way over 100k but not 1 million USD.
Support tickets went unanswered for months, assigned account reps left us hanging for months with multiple follow ups, etc.. All tickets opened within the last 6 months got AI generated responses with massive delays that indicate the ticket wasn't read by a human due to how inaccurate the response was based on the questions asked.
I had an obscure KMS issue a few months ago, I reproduced it and opened a ticket. It was a low severity. A guy emailed me back in about 4 hours, acknowledged it and thought I was doing it right. Came back 2-3 days later noting that it was a bug, and would be fixed in a few weeks. As a workaround he updated my script with a less obvious/efficient solution to get me through.
Two things can be true.
We literally pay hundreds of thousands a year in Oracle support contracts, and this is what we get? AI bots? Nope. Migrating to Postgres is now a top priority.
This "replace people with AI" nonsense has to stop. [0]
[0]: https://www.forbes.com/sites/jonmarkman/2026/04/06/oracles-m...
This enforced adoption of immature GenAI reminds me of Milo Minderbinder trying to make people eat cotton in Catch 22, because he had inadvertently obtained a huge amount of it.
there’s no heroic clash with other civilizations, they simply rot away
Remember the “Prussian general 2x2”? https://quoteinvestigator.com/2014/02/28/clever-lazy/
AI enables the stupid and diligent. When I get long form emails from people I don’t know well, I assume that 80% of the time it’s AI bullshit. People are having AI respond with bullshit to other people’s bullshit.
I’m actually deliberately adding bad grammar to communications as a hook that you should read it. It hurts my soul.
I still don't have an explanation why people are doing this. Is it part of leadership training? Or do presenters have their own theory that including this stuff makes the presentation more memorable and enjoyable?
Why was leetcode so popular? Because Google did it and they were the cool kids at the time.
There were always other problems too, pressure on the company in both directions across many different product lines on both cost (any number of cheaper baremetal providers who are much faster at providing customers instances than they were a decade ago), and product quality (any number of startups to now bigger companies, databricks probably being the biggest success) along with a number of expensive bets that were made that didn't work out especially as interest rates began to rise (there were numbers of of different services ranging from IoT, AI, business support, robotics, groundstation, that essentially all failed).
AI infra being their latest bet, along with doubling down on custom hardware is smart, but these roles don't require the same number of SWEs and instead require a different type of high skilled professional.
Unrelated to your main point, but it's "alumnus" in the singular form. For bonus language nerd points, you would use "alumna" to refer to a woman, or "alumnae" to refer to multiple women. Not sure how Latin handles mixed gender groups, though I would guess it's "alumni".
I personally think it’s not worth it.
And things only got better post-Industrial Revolution when labor organized and forced the issue.
There's no guarantee that will work again if labor has reduced leverage due to AI reducing their value.
I think in one way or another this all works itself out, but I'm not convinced it won't be a very painful (and possibly violent) transition to whatever comes next.
A software engineer getting four months severance after a layoff exists in a different universe from this so no. There is no precedent. Don't you dare talk about the industrial revolution because its not even in the parking lot of the ballpark.
You're right, it's not in the ballpark. It's at the gates. The game isn't on simply because they poisoned the opponent in the duggout.
It's time. It needs to stop now while the body count is low. This isn't about some dev getting severence. They've taken away the street sweeper position and are watching us eat each other.
There are paralles to the industrial revolution, but it seems the working class is cut out from being able to deny labor in exchange for better conditions.
It‘s the golden age for software engineer employers.
The reason is simple: nobody wants to bring in toxic culture.
It has become extremely difficult to hire at any level, we have had an open position on my team for a senior data scientist for a year and a half now, with barely any candidate applying, and none of them being competent.
Similarly the average level of new employees has dropped dramatically. The famous "hiring bar" is now below ground.
Since then I've had no less than thirty five recruiters contact me for a systems engineering role at Kuiper.
Even though I'd "only" be a yellow badge (contractor) every time I've hit the Amazon hiring manager desk I get instantly vetoed.
Amazonians don't know (and I guess we can never know?) how long we're blacklisted for.
Eventually they're going to run out of people
Limit scale. Enjoy your craft. Become immune to hype and "what's popular," instead focusing on "what otherwise inaccessible experience can we make possible for our customers?"
When you let the money guys take the wheel (not the "passionate nerd" types), it inevitably (and I would argue necessarily to keep the lights on for such a big org) results in a shift to spreadsheet brain.
If you haven't seen it, watch Jiro Dreams of Sushi and see if that way of life resonates. Also check out his interview with René Redzepi from Noma. Lots of great insight into how focusing on your craft implicitly creates the opportunity for creating and delivering great things to others.
Not sure how to address it though. I suspect keeping companies small and focused on quality, sustainability, and free of VC influence would be a solution. It'd take continual work though, like tending to a bonsai.
I also joined in 2022, and it aligns so much with my experience. Good manager that moves on, then a gradual erosion of "insist in the highest standards" towards a dreaded "good enough", GenAI only accelerated it IMO.
I agree with you that it's up to the people to adopt new tooling so they don't get left behind.
But from the company perspective, it's beneficial to get people using the new tools as fast as possible so that the overall impact on end user is accelerated.
An example is the adoption of CNC machines in factories. If you're a old timer using mill and lathe, you may argue that the company should not enforce all its workers to learn how to use CNC machines because you can still use the lathe and mill just fine. But from the company perspective, having you learn and run 3 CNC machines at the same time is much better than have you run 1 mill or lathe.
Meta can get its Sysiphus on all it likes; whether or not genius pours forth remains to be seen.
"Fungible" implies they are a commodity, easily swapped for someone else. In other words, they are so low-value that they are interchangeable.
"Flexible" or "generalist" instead connotes that they are so high-value that they can operate well in multiple domains, easily shifting to where they are needed most.
You can easily call the typical Japanese life-long employees as “flexible” or “generalist” but not an employee of a company with median tenure rate of 1-2 years. That’s fungible.
This is how many large enterprises still operate today. Ironically, the main argument is that it's faster to provision VMs on-prem than it is to get approval to run in the cloud.
Bureaucracy always beats tech.
I bet as the managers publicly nodded in praise for his heroic act, their hands were already typing his name to be sent to HR for “get this guy out of here on any excuse you can” note. (In reality it would be a nonverbal hint of sorts. Nothing to leave any trace discoverable by lawsuit)
> It also assumes that there is a limitless supply of people with the required skills, and a willingness to work for Amazon.
I think one major concern here is that "apparently" AWS is in such good place they don't have to worry about anything. People, their reputation, their employees, future stability, growth. Nothing is of worry. They have reached the IBM status where everything is awesome. However, executives are usually paid precisely to worry about these things. Though, we've seen executives are pretty stupid these days. So, I guess it's clear AWS is clearly not the exception.
And this is not a dink on the ai tooling itself but on the organizationan processes that provide the context in which the AI code generation is being used.
Bad processes will always produce bad low quality outcomes regardless of tbe technology.
The goal was never to solve a real problem, like we evangelized for decades. That was how it was explained when resources (mainly time, but also money) were scarce and we could not just throw things at walls. Now we can, and you won't see anyone talk about "make something people need".
Things will be low quality until something sticks, and then money will be poured into it. It's not a bad strategy, but my takeaway from this is: there are multiple plausible explanations for the same thing. People have an incentive to not give you the correct one if it helps you compete with them. But they will give you a sensible one. AI won't protect you from this, experience and real knowledge will.
AWS has been this way for a lot longer than GenAI, since the basic infrastructure products were built out early on. But when I read this line about throwing things out there quickly, I also think of Google and even Anthropic. Google has a long list of products that got created and killed, as part of their internal politics and promotion culture. Anthropic is currently rushing vibe coded slop all the time to try and win over OpenAI and set up their IPO.
Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.
> To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there. Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments.”
Both are following the same strategy. Amazon has a $2.86 trillion market cap. That's the equivalent of 143,000 $20 million Series A startups. Companies like Amazon and Google are basically an integrated herd of cash cows plus a VC portfolio.
Has that changed, or is it the non-AWS part of Amazon?
Anecdotally, this seems to be the new "mission statement" of many companies.
Many storied companies can be described this way. It’s a shame. Have any companies hit such scale and kept the ethos and magic of before? Is it inevitable for companies to enshitify themselves in the pursuit of their shareholder’s goals?
If any of you young'uns read this, that is not how we had to do provisioning before cloud.
VMs already existed before AWS came out. You could already provision a new server usually in minutes and rent it month to month.
In fact, all the existing VM server companies had to start calling themselves cloud companies because pointy haired bosses couldn't understand what cloud really was.
Where could you rent VMs in 2006?
IIRC there were two ways to run stuff, get your own server or get an account on a big shared computer.
And they were cheaper than renting AWS. MUCH cheaper. They still are.
The original point of AWS is that could scale according to demand. Have 10 VMs running at lunchtime and 1 VM running at midnight.
But using a cloud VM also required less server admin experience. It was a bit easier and came.pre-configured with things like firewalls.
And THAT is what ended up being the USP of cloud hosting. Especially when they started rolling out all the SQL as a service, redis as a service, etc.
You didn't need to really understand servers to run a server, and it turned out almost all developers really didn't want to understand servers. TBH, I don't, server admin sucks. Right now I'm working somewhere where I have to think about SSL certs occasionally and I consider it a complete waste of my life.
Digital Ocean came out like 5 years after AWS, what was revolutionary about that wasn't that you could spin up VMs quickly, it was the price. VMs went from $20-30 p/m to $5.
For developers who weren't SV rich, that meant you could run a side project without it being a significant cost.
Linode definitely had something along those lines.
Amazon won on APIs and overall integration but VMs were around already.
I remember the story really well as this is when i joined the workforce as a young GNU/Linux fan.
Also people are throwing AWS start date of 2006, but AWS only really started catching on around 2008/2009 if I remember correctly. EC2 came out of beta late 2008, and EBS was only launched around the same time.
I don't think it even officially launched S3 until 2008, which is what all people really used it for initially for cheap remote backups.
It definitely took a while to get good. There was also a period where having 'noisy neighbours' impact VM performance was often discussed here. You didn't tend to have that same problem on the other VM hosts as people were using the VMs for hosting with generally low CPU usage, not for compute.
I was renting a Debian VM from Bytemark in 2005 that I used to host a mail and web server. I think they were one of the first operators in the UK.
The recovery from being an orphaned customer account serves as the litmus test. In this case, it took someone who was unique and non-interchangeable and poked "the right bear" -- and it succeeded. But that's precisely the way that enshittification of the principle occurs.
Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die. With that said, due to its late-stage capitalistic ethos, Amazon is just too overly gleeful about this tasteless reality of life. It's the equivalent of a nephew coming to an aunt's funeral and shouting "A week ago, I told her everybody dies! And now she did! Wasn't I right??? Everybody dies!"
> Also, last year the focus at AWS turned fully and almost desperately toward GenAI.
I wonder if I'm being too cynical, but late-stage capitalism companies also love profiteering, and the mere prospect of firing all those pesky workers and not having to pay their salaries is like cocaine to those organizations. Which is why I think Amazon fulfillment centers will at some point rent robots at a price point between 2x and 3x their current human labor costs, in the hope that it will eventually make economic sense.
Enterprises typically have this mindset. Most corporations I worked for in fact treated employees exactly like this.
As for needing this mindset, I am not so sure. There is a spectrum in between going under because a storied employee retired and treating employees as meaningless numbers in a spreadsheet.
But ultimately I fully agree with the whole of your post. I just had to nitpick about this.
Side note, and unpopular opinion ahead: while it takes a lot of courage to write things like this and I respect it, but being fired and writing negatively (no matter how justly) about your former employer is considered by many employers as a red flag and can hurt you going forward (even if you are 100% right).
No they utterly failed and needed a special non fungible employee to get them to do their job.
I'm glad to see that one core amazon principle has endured the 10 years since I worked there, even if none of the actual leadership principles have survived /s
It’s well into the IBM phase now. Primarily providing important but boring commodity infrastructure, but the top talent that can drive real innovation has long since left the building.
It’s race to stay relevant in AI but always seeming 2-3 steps behind everyone else is one such example of the current sad state of affairs.
https://aws.amazon.com/executive-insights/content/how-amazon...
AWS services still are generally reliable and available. I’d think we’d be seeing cracks here if the organization were in shambles. AWS seems to keep humming along.