AWS Tools Suck | Hacker News Reader

AWS Tools Suck

cyclic.sh

118 pointsheldsteel772 comments

> This is not a case of “not enough resources'', this is a case of the project not getting the internal priority to invest in improvement.

This is only partly true. Fun fact: Amazon is so big and the churn so high, that hiring talent has become the bottleneck.

So yes, parts of Amazon have prioritised new/more features over improving existing ones, and some are drowning in a sea of legacy. But also parts simply don't have the number of experienced engineers they need to deliver all their goals.

Disclaimer: I work for Amazon.

> churn so high, that hiring talent has become the bottleneck.

It was a matter of time for that to happen. Amazon is infamous for having a firing quota, and infamous for Hiring managers to hire people, to fulfill said quota.

As long as these practices remain, the situation will only worsen for Amazon

Top performers leave the company voluntarily, most often to get a promotion and pay rise elsewhere. This is commonplace in tech. The problem is (particularly during the last few months) the intake of talent has struggled to keep up.

There is no firing quota.

According to multiple sources, Amazon has a policy to have 6% UnRegrettedAtiration. which in normal speak is 6% of employees to be fired per year. and managers are expected to fullfil this quota, and backfil it if they failed to meet it previously.

Here is a recent news mentioning it: https://www.arkansasonline.com/news/2021/jun/27/amazons-unre...

Nuance, but you can have someone leave you don’t regret without firing someone.

Sure, but let's say that the quota is not met organically and only 3% leave voluntarily. Then to achieve the target 6% unregretted attrition for an org, you'd necessarily need to pick a different 3% to fire/push out.

to make this clear: as a SDM at Amazon who is probably not really authorized to speak about this - there are goals but there is no quota that says "you must fire your bottom 6%". At least, no one has said to me you must fire someone. That's just nuts. At any organization there tends to be some under performers so if you are doing your job the problem sorts it self out.

> There is no firing quota.

It's called planned attrition.

> Fun fact: Amazon is so big and the churn so high, that hiring talent has become the bottleneck.

This is only true because Amazon refuses to pay market rates for talent. E.g. for the folks who write their technical documentation, they require that they have the technical skills of senior engineers, but pay them closer to what you would pay a recent college grad to churn out SEO blogspam. If they actually payed their technical writers the same amount (or more) as their engineers, their documentation would be great. They have the money to do so, they just refuse to actually do it, because they're big enough that they can externalize the costs of bad documentation onto their customers.

I can only talk for Software Engineers. Amazon pay for engineers is by no means low, definitely above the median. But it doesn't pay top dollar either. We are constantly losing good candidates to other companies for a higher pay. That's not counting those that don't even bother applying, for which we don't have visibility.

Come on mate. If you compare every other employer on earth yes Amazon comp looks great but limit it to say FAANGs and it gets interesting. Even more when you look at vesting structures (even worse that even your 401k match vests after three years. Or that your refreshers are tied to what the current value of your past grants are etc!

Why would anyone work at Amazon except for desperation?

The hilarious thing about your "raising the bar" hiring is that the abusive hiring process, abusive work culture, abusive HR policies, and the rapidly declining reputation cap the bar. Why would good people work at fucking Amazon? They won't. So you'll end up with a bunch of backstabbing Machiavelli wannabes.

And those people that shake out of Amazon? They don't have tech talent to rise above the machiavellis, so that means they are the FAILED machiavellian backstabbers. What could be worse? That means they will be ULTRA machiavellian out of paranoia in the next workplace.

Sure, tell yourself "this happens everywhere". "It depends on the team". I have never met an amazon employee that didn't admit the place was a terrible workplace and they only worked there for the resume checkmark.

Usually this pathetic machiavellian game theory plays out in middle management. Amazon however has trickled this down to the tech workers, which is not good.

All that for "ok" pay? Your vesting is a three year carrot that the HR policies try to ensure that you won't be able to cash? What a crap company.

Disclaimer: I was recently hired into AWS.

I’m not seeing the kind of organization that you’re describing. There may be parts of Amazon that work that way, but I don’t see it. Maybe that’s an Amazon versus AWS thing, or maybe not.

I would say that any company with 1.6m employees is going to have radically different policies and procedures in different parts of the company. Maybe you’ve only heard about the bad parts so far. I see this as the same issue as the future being distributed irregularly.

We do have a hiring issue in our team, where we are short by dozens of SDEs that we wanted to have by now. And we are definitely still hiring.

I can’t speak for anyone else, but my incentive comp is not backloaded, and I feel like I’m definitely getting market rate pay.

I don’t consider myself to be desperate. I was already on my way out the door at Whole Foods when I virtually ran into a guy on one of the internal channels, and he thought I’d be a good fit for what they were looking for. I interviewed, and I got the job. But I think I could have gotten a job at a lot of other places, too.

@AtlasBarfed — There is no part of what you’re describing that fits what I’m seeing in our team or what I can see of other teams.

But I can only speak for myself and my own personal experience.

Your company has a stated policy, repeated all over the internet, or a 6% yearly desired attrition rate.

Does your team have to align with that? It does not sound like it.

Therefore, your team is an exception to the HR rule. Sounds like you are not typical then.

And this is not an issue of HR policies applied to the "lower classes" of Amazon factory workers. These are HR policies applied to the developer class.

Why are you so short of dozens of SDEs? I would say that is ... 30 or more? Because you can't hire them through your LUDICROUS hiring process? Because you've "desired attritioned" so many? Why are you so short of developers? Ask yourself that.

How old is your team? What is the tenure of your coworkers? What is your turnover? How long have you been at your job? How many developers have you seen turned over? If you're short "dozens" of people, how does your team function? Do they just overwork you? If the team is functioning with 24+ shortfalls, why would Amazon fill them?

Yes, working everywhere in the modern world is all corporate shit to some degree. Maybe you feel comfortable now, but your team can get reorg'd. Your boss may move on, and you'll get a new boss. Then all of a sudden a tech problem makes you look bad.

I guess some people work for shitty companies because it makes their cv look "good"? And the pay is probably decent too.

But personally, never ever working for a big American tech company. They have no soul.

>But personally, never ever working for a big American tech company. They have no soul.

Bell Laboratory's was a thing....

That's because T.N. Vail was a visionary leader. They are in short supply nowadays.

True words!

Hot take

If that's what your interested in go work at Tesla or SpaceX.

If Amazon engineers said they never work more than 40 hours a week and shut off their work phones at 5, I’d consider their pay decent.

And maybe it’s true. But I’m not sure.

If that’s true, perhaps they should reconsider a) their non-compete agreement b) their equity vesting schedule.

Long ago AWS reached the point where feature prioritization was discussed in terms of millions of $$ in revenue. When you have CTOs at Fortune 500 companies waving 10s of millions of dollars in front of your face to build a feature, it's much much easier to prioritize building that over one that is requested by smaller developers with uncertain revenue impact.

> The only conclusion is: bad tooling isn't affecting their sales.

I actually think bad tooling does affect their sales and long-term growth, but that they're blind to it because they can't easily measure it, and they have an obsession with data-driven decision making.

I suspect this is also exacerbated by hiring problems due to their reputation as an employer, forcing them to make tradeoffs that sacrifice dev UX more often than they'd like.

Shouldn't a company of that size and revenue be able to do both at the same time?

I have absolutely no evidence to back this but I think they are actually making money of the bad tooling.

Firstly, because it requires and also creates an army of cloud AWS consultants that works as free sales persons for AWS.

Secondly, this army creates overly complicated and expensive solutions which then only they can change, and they never do because it looks good on CV.

Maybe a bit too cynical.

> Shouldn't a company of that size and revenue be able to do both at the same time?

On its own, probably, but holistically it’s always more complicated than that. They don’t have infinite resources and zero coordination costs, so they still have to make product development tradeoffs.

> I actually think bad tooling does affect their sales and long-term growth,

Anecdotally, I know a fair amount of companies that gave up on using AWS native stuff and went to EKS for the K8s tooling and ecosystem.

The old problem of data-driven decision making: you tend to only measure success with easily observed metrics. # of features launched is the new # loc.

Data driven design is exactly why their UX is so awful. Give it a few years and someone else will be able to compete just by offering a unified view of deployed resources.

Data says no easy single source of truth- must hide everything in submenus for true kafka maze

Yes they might suck, but that's because of its success and the complexity of the problem - it'll get better over time.

However aws-sdk doesn't need to be 70mb, the console doesn't need to be slow and they don't need over 10 unique services to deploy containers.

We're working on a much improved dev ex for cloud providers and Kubernetes at https://northflank.com - with a fast real-time console, a well documented and useable API with auto generated CLI and API clients. 30 seconds of fun - can I get a production ready CI/CD setup for a repo(s) in 30 seconds? Can I get a HA Redis, Postgres, RabbitMQ provisioned and connected to my code in 30 seconds? These are the questions and solutions that developers will be asking for more and more.

Your product looks really cool, but just a heads up as a potential customer:

On your cloud >> talk to sales

Every SaaS firm on earth wants me to talk with their sales, I'm tired of that. From 2022 and beyond, if you can't give me some Databricks style onboarding, you're not getting my money.

Hi pid, thank you for the kind words. Agreed that's not our intention. You can signup self-serve at the moment and get access to the entire platform + free developer tier with our infrastructure. On your cloud is currently a manual on-boarding with our engineers, in the new year it will be also self-service.

I don't find the console slow, but I find it pretty stupid. There are some things that I can't do via the console, only via the API, and some stuff just doesn't make sense. Like why does AWS WAF have its own region selection (that always defaults to us-east-1) separate from the console region selection?

>Like why does AWS WAF have its own region selection

Assuming you're talking about WAFv2, it seems that's because they chose to have a single set of api endpoints instead of regional-based endpoints. And that the CLI --region option is really only equipped to swap endpoints.

Doesn't explain why they chose that path though.

I think even WAF v1 had stupid region selections on the console.

Actually they can't easily fix their mess of tools because they have so many customers vendor locked into them. Vendor lock-in is actually the primary reason for the tools sucking so much: it's a feature not a bug and it's intentional.

It prevents customers from leaving and it prevents customers from lowering their amazon bills by making more effective use of what's on offer. Amazon makes it really easy to spend money on their platform but very hard to save money by simplifying or more effective use of resources. That is their business model. They always offer an easy and expensive and slow/convoluted, slightly less expensive path for doing anything.

If it was easier, countless companies would be saving a lot of money and Amazon's profits would be decimated. Or worse, they might be tempted to jump ship to a competitor. The primary goal of complexity is to keep customers after they buy into that.

Of course, over time it has exposed them to companies trying to compete with them with a better developer experience. To mitigate that, they invest continuously to ensure those companies never quite catch up. And of course that keeps on adding complexity. Which is good for them.

Most big software companies work with the same business model. IBM, Oracle, Salesforce, SAP, etc. it's all about vendor lock-in through complexity with those companies. It's how they make money. Lure customers in with whatever they need, sell expensive stuff, consultancy, training, etc. and before you know it you are in a decades long relation ship with your customer where you basically cream off their revenue on a monthly basis. AWS is here to stay. However, that doesn't mean it is the smart thing to be using right now for small new companies.

I don’t know, maybe I’m In the minority, but I quite like the tool chain around AWS.

AWS CDK is a breakthrough in infra as code. It will be the standard in a few years to come.

AWS SDK v3 is quite ok and will get better soon.

AWS CLI is maintained and has everything well documented.

Most popular services work well together.

IAM is very powerful and a well thought out solution.

Same. They need a lot better tooling around lambdas etc, but CDK is really nice, especially when compared with terraform

A big issue is lock-in, which things like k8s help mitigate. Another is price opacity, they’re the most expensive and the hardest to budget for. There is a cottage industry around helping large corps understand and plan for AWS charges.

> Reason 2: AWS doesn't sell based on Dev Ex

AWS's core is developer experience.

AWS is an API. Anybody who does not realize or can not exploit this fact this pays massive premiums for using it (probably a lot of businesses which should have no business using AWS directly).

The application tooling is extra. And as far as I can tell, AWS is the only major public cloud that has decent coverage by tools of any flavour against their API, and they have the most decent set of language SDKs available. I am not familiar with GCP but anything Azure puts on the table is a catastrophe when compared to the ease of using boto3 to get a certain flow working.

Admittedly, their console has some defects it ought not to have, but with some systems and software development knowledge, you can grok all of the tooling AWS releases pretty easily.

The biggest of my problems with AWS that besides their API, some of their managed services are just not on the quality level as their core services (ec2, s3, ...).

GCP is pretty pleasant overall. The API, command line, and UI are all pretty good. Vast majority of functionality is supported in all three places too. Like all providers, they have a few services that suck but overall Google has done a great job at developer experience.

> AWS is an API. Anybody who does not realize or can not exploit this fact this pays massive premiums for using it (probably a lot of businesses which should have no business using AWS directly).

It's a freaking huge API and all of us would suck a little bit more of it without tooling around it.

> AWS is an API.

Exactly.

Most of my grief is wrestling with the misc tools which obfuscate the underlying API. Bad or good design aside, I just need to see what's on the wire. Once I kinda grok the mechanics, I can map whatever tool's (terrible) UX to the reality.

It's my mental quirk. Maybe the inability to suspend disbelief. It's always slowed me down. But once I have a mental model, I can move pretty fast.

My favourite example is that if you have insufficient permissions the CLI may just give you a detailed but completely random and incorrect error. Eg if you are setting MFA Delete but you are not root, the error message it displays is the one for Dev Pay which is a nice way to waste users' time.

Eg https://stackoverflow.com/questions/45602558/devpay-and-mfa-...

Even their console has really basic bits of weirdness and friction. Like if you want to view the logs from a fargate container running in ECS (possibly one of the most boring things you can do on AWS): * For some reason, the log viewer within the ECS console has worse filtering capabilities than the Cloudwatch one * Attempting to view logs for an instance which crashed more than a few minutes ago is very risky - sometimes the last ones are either missing completely or very hard to find in the UI (Cloudwatch will have them though!) * They tried removing it in their ECS console revamp briefly

I got bitten recently by AWS documentation. Several pages strongly imply something to be true, which in practise was 100% false. If I'd wanted to make a living reading text between the lines to work out how they're trying to fuck me, I'd have gone into law.

Also their docs suck at saying the limitations of their services explicitly in a transparent way.

Please do give feedback if you see anything inaccurate on the documentation. I've submitted feedback many times and they would sometimes reply to my email that they have updated the documentation.

I've opened pull requests that haven't been touched by AWS for ~9 months (comments or other). Been told (through a back channel) that the team didn't have the capacity to review all the PRs.

So "it depends"

It only has to be as good as it's competition. Haven't used GCP, but would pick AWS over Azure when it comes to tooling (and a few other categories). AWS documentation tends to be better - for example, Azure fails in giving quick example output for CLI commands. The AWS Console is better than the Azure Portal.

I haven't used Azure, but I'd pick GCP over AWS any day. Tools are great, things fit together and feel consistent in a way AWS never did. Feels more like a unified approach to a holistic cloud product, rather than a bunch of teams fighting each other to ship features to gain market share.

Same with GCP: their current maple makes AWS look like the pinnacle of fast web applications, they keep “accidentally” breaking it for non-Chrome browsers, and the CLI is full of ideas which you can see are kind of appealing but someone got distracted and moved on before finishing the hard parts. Then you get some of the default security policies you’ll be cleaning up for years…

This is not to saw AWS is perfect but it’s not like people are inexplicably using a bad option. We still have a lot of maturing to do as a field.

> Reason 2: AWS doesn't sell based on Dev Ex

Hard disagree here. The reason why AWS can get away with shoddy tooling is that the competition is orders of magnitude worse. Azure's web UI is an unmitigated desaster, and while I never used GCE I can say I won't ever use it for the simple fear of some "AI" running amok and killing off my Google account - too many horror stories here on HN for my taste.

I'm quite surprised at this blog post, since AWS has popular tools like CDK that make creating cloud resources developer friendly, while leveraging familiar programming languages and high level constructs. A lot of the partner and 3rd party options are more targeted (imo) for supporting multi-cloud use case with a unified tool and to provide an alternative to AWS native tooling.

Disclaimer: I work at Amazon.

I agree with points 1 and 2, but not with 3.

AWS isn't a developer tools company. It is ops tools company. In particular enterprise ops tools company. Their customers are IT managers and system administrators from large companies. That explains 1 and 2.

Famously, AWS is organized as huge bunch of two pizza teams. Essentially, it's a huge incubator for internal startups. That's how they manage to churn out new features so frequently and try out and discard unsuccessful products. Also, that is why their tools looks so damn inconsistent and why you never know what's working with what.

Regardless of money, they can't make the tools better without sacrificing something. And that is a space for competitors. Work on developer centric tools for small and medium sized companies.

As I harped on in the AWS outage, when you combine this fact with the disruption of Stack Ranking/forced attrition, how does this lend itself to long term stability for something that is approaching utility-level importance for the economy?

Utility companies are staid and boring. That's a GOOD thing.

If AWS doesn't restructure, then what does it do? Re-re-reimplement all the internal bespoke systems that run AWS? Doesn't matter, their employees will turn over and the bitrot in those systems is about, what, four years?

Man I hope they are good at microservices!

I think there are a few things going on. First, it's tools designed by people who work at big companies. So the tools are insanely powerful, but there's high overhead to use them. Simplicity matters a lot less

Second, there's an arms race in cloud infra. So it's more about adding functionality, ticking the box, being ahead, than being simple and usable.

Finally, frankly, poor design. The AWS console is a usability nightmare.

That said, AWS is awesome. It's just infuriating to use if it's not your day job and you don't know it in and out and aren't willing to spend days reading their docs.

And still the and miles ahaead of "user experience" than their competitor (they are pretty much unworkable for developers, it's insane how bad that ux is).

Have been an AWS user by default since launch. Basically try to muddle through, have had some good experiences (RDS, S3, EC2, Amplify hosting) and some frustrating moments where I threw up my hands (other parts of Amplify, doing custom stuff with Route 53).

Now I'm using Firebase and Google Cloud Storage on a new project, and I've generally found the UIs to be clearer and better-organized than AWS, and the documentation more accessible and easier to understand.

These GCloud tools really do feel tailored with the average hacker in mind, something I often don't find at AWS.

UI/docwise, AWS kind of expects you to to come to the mountain, not the other way around.

As an example, getting Firebase Auth up and running was worlds easier than Amplify Auth, which I flailed helplessly at and gave up on.

Curious to learn more about GCloud, see if the rest of it is this good.

> If the decision maker cared about developer experience why do they keep choosing AWS?

If you have to go to the cloud what else is better. Google, MS? If you press hard enough the same amount of shit drops on your feets.

Catchy title that's sure to gain interest with the hate-everything-Amazon-does crowd. But the only tool they point out is IaC.

CloudFormation works in both YAML and JSON. Don't like that? Use the CDK to write in your favorite language and that will be transformed into CloudFormation template. I'm not seeing what sucks here.

The reason most people use Terraform or Pulumi is because they want something that's cloud agnostic. That doesn't mean CloudFormation itself sucks -- it means that people have a different usecase.

Was just playing with Amplify Studio and have no clue who it's targetting and who will be happy and for what use cases.

So how does this work? Features are delivered then another team adds capabilities to the cli/console?

Say the eks team responsible to deliver a cli/console tooling?

It's an interesting problem to solve. Any insight on how to go about managing this is greatly appreciated.

Uh... what?

Have you tried the serverless tooling from GCP? doesn't hold a candle to AWS SAM.

Comparing Aws tools to Terraform and Pulumi doesn’t feel fair. Compare against Azure and Gcloud. IMHO all cloud devops tools kind of suck, third parties included, but man do they cover a lot of complexity

AWS once tried/looked at/toyed with buying my company, mostly on the basis of the 'developer experience' we had created.

In the end they didn't buy us and the aped everything they learned and launched a carbon copy product 18 months later. rages

... but...

one thing that caught me off guard during the process was that their teams had no idea about developer experience, didn't understand workflows outside their own domain, and didn't pretend to understand any of it.

Honestly the only card they really had to play was being aloof and condescending. Of course we were very deferential to them the entire time, because AWS.

This was one group within what is a massive organization, so YMMV, but if they come knocking my suggestion is to be aloof, kind of a jerk, and don't share anything that isn't on your www. My guess is that would really be what gets their juices flowing.

IMO in this case the documentation is what sucks. AWS works if you know how. The problem is precisely that, make it work.

Does this article not need to say more than the <title> and a picture of tools? Because that's all I see

They do suck.

They also have everything.

As long as there are other companies building on top of AWS, I’m pretty happy with them.